NetCDF build fails with test failure because of ftp not reachable

[aafaque33]

• netCDF version: 4.7.4
• OS Version: Rocky Linux 8.10 (Green Obsidian)

Build / Installation of NETCDF4 from source fails because of a test failure that seems to related to ftp downloads.

Searching on issues / changes found it was because recently recently the ftp site has been archived and recently switched from ftp to https download in master branch which is not part of any release even latest 4.9.2 (The build / installation was for 4.7.4)

• changes associated with the removal of the Unidata ftp site. #2915
• ftp --> resources, part II #2924

Below is configure command and then make command

> CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure --prefix=/usr/local --enable-hdf4 --enable-hdf4-file-tests

> make -j check

Output

make[2]: Entering directory '/var/tmp/appeears-source-builds/netcdf-v4.7.4/hdf4_test'
make[3]: Entering directory '/var/tmp/appeears-source-builds/netcdf-v4.7.4/hdf4_test'
PASS: tst_hdf4_extra
PASS: tst_h4_lendian
PASS: tst_chunk_hdf4
PASS: run_formatx_hdf4.sh
FAIL: run_get_hdf4_files.sh
============================================================================
Testsuite summary for netCDF 4.7.4
============================================================================
# TOTAL: 5
# PASS:  4
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

hdf4_test/test-suite.log

   netCDF 4.7.4: hdf4_test/test-suite.log
============================================

# TOTAL: 5
# PASS:  4
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: run_get_hdf4_files.sh
===========================


Getting HDF4 sample files from Unidata FTP site...
Getting HDF4 test files AMSR_E_L2_Rain_V10_200905312326_A.hdf AMSR_E_L3_DailyLand_V06_20020619.hdf     MYD29.A2009152.0000.005.2009153124331.hdf MYD29.A2002185.0000.005.2007160150627.hdf     MOD29.A2000055.0005.005.2006267200024.hdf
./run_get_hdf4_files.sh: line 22: wget: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: ftp.unidata.ucar.edu
wget failed: try 1
./run_get_hdf4_files.sh: line 22: wget: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: ftp.unidata.ucar.edu
wget failed: try 2
./run_get_hdf4_files.sh: line 22: wget: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: ftp.unidata.ucar.edu
wget failed: try 3
./run_get_hdf4_files.sh: line 22: wget: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0curl: (6) Could not resolve host: ftp.unidata.ucar.edu
wget failed: try 4
Could not ftp AMSR_E_L2_Rain_V10_200905312326_A.hdf.gz
./run_get_hdf4_files.sh: line 47: return: can only `return' from a function or sourced script
FAIL run_get_hdf4_files.sh (exit status: 1)

Question is how can we avoid the ftp failure or have the latest https download in the previous releases as it seems like only in main branch?

• https://github.com/Unidata/netcdf-c/blob/main/hdf4_test/run_get_hdf4_files.sh
• https://github.com/Unidata/netcdf-c/blob/v4.9.2/hdf4_test/run_get_hdf4_files.sh

[WardF]

At a glance, the issue appears to be that wget is not installed on your system; what happens if you install wget and try again? The url should redirect properly otherwise.

[aafaque33]

I have got wget installed and it still fails. Just to mention again the error is for version 4.7.4 (similar for 4.9.2) where https://github.com/Unidata/netcdf-c/blob/v4.9.2/hdf4_test/run_get_hdf4_files.sh uses ftp instead of https and ftp seems like not reachable now.

Full installation commands for source

> curl -L "https://github.com/Unidata/netcdf-c/archive/refs/tags/v4.7.4.tar.gz" -o "netcdf-v4.7.4.tar.gz" 
> mkdir "netcdf-v4.7.4" 
> tar -xvzf "netcdf-v4.7.4.tar.gz" --strip-components 1 -C "netcdf-v4.7.4" 
> cd "netcdf-v4.7.4"
> CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib" ./configure --prefix=/usr/local --enable-hdf4 --enable-hdf4-file-tests
> make -j check (Fail here)

[WardF]

@oxelson it's my understanding that ftp redirects to the appropriate url; am I incorrect?

[oxelson]

No, we cannot redirect for the ftp protocol.

[WardF]

@oxelson Thanks!
@aafaque33 given the age of the release you are working with, the easiest thing to do may be to go into the shell script which is fetching the hdf4 files and make the appropriate change there to the url that is being accessed. The relevant line is in the getfile() function, and you should be able to replace what's there with:

   DATAFILE="https://resources.unidata.ucar.edu/netcdf/sample_data/hdf4/$1.gz"

Hopefully this helps!

[aafaque33]

@WardF That i can do, but I was wondering if the fix for https url will be in an upcoming release soon ? as I can see it in main branch only
The reason I am using the 4.7.4 release as our docker container was using it previously, but currently also thinking to update it to 4.9.2 but even that has ftp so seems like even update of netcdf4 version will not help currently/

[WardF]

Yes, I am working to get the first release candidate for 4.9.3 out as soon as I am able, hopefully by end of day or first thing tomorrow, with 4.9.3 to release shortly thereafter.