Unable to use absolute file path on mingw64 with NetCDF 4.8.1

[Alexander-Barth]

• the version of the software with which you are encountering an issue

NetCDF 4.8.1

• environmental information (i.e. Operating System, compiler info, java version, python version, etc.)
  Window 10, mingw64 gcc (version 10.1.0)

• a description of the issue with the steps needed to reproduce it

The C file below works with NetCDF 4.7.4 but it produces an error with Error: No such file or directory with NetCDF 4.8.1.
I am debugging an issue for the Julia NetCDF package which today upgraded to 4.8.1. Linux and Mac OS X are fine.
In Julia, all C packages are cross-compiled from a Linux host. I was able to reproduce the issue indepdently of Julia.

This file is the sample simple_xy_wr.c file with an adapted filename to show the error.

#include <stdlib.h>
#include <stdio.h>
#include <netcdf.h>

/* This is the name of the data file we will create. */
/* fails in 4.8.1; works in 4.7.4 */
#define FILE_NAME "C:\\Users\\ALEXAN~1\\AppData\\Local\\Temp\\foobar.nc"

/* works 4.8.1 */
/*#define FILE_NAME "..\\AppData\\Local\\Temp\\foobar.nc"*/

/* works 4.8.1 */
/*#define FILE_NAME "Test\\foobar.nc"*/

/* We are writing 2D data, a 6 x 12 grid. */
#define NDIMS 2
#define NX 6
#define NY 12

/* Handle errors by printing an error message and exiting with a
 * non-zero status. */
#define ERRCODE 2
#define ERR(e) {printf("Error: %s\n", nc_strerror(e)); exit(ERRCODE);}

int
main()
{
   /* When we create netCDF variables and dimensions, we get back an
    * ID for each one. */
   int ncid, x_dimid, y_dimid, varid;
   int dimids[NDIMS];

   /* This is the data array we will write. It will just be filled
    * with a progression of numbers for this example. */
   int data_out[NX][NY];

   /* Loop indexes, and error handling. */
   int x, y, retval;

   /* Create some pretend data. If this wasn't an example program, we
    * would have some real data to write, for example, model
    * output. */
   for (x = 0; x < NX; x++)
      for (y = 0; y < NY; y++)
	 data_out[x][y] = x * NY + y;

   /* Always check the return code of every netCDF function call. In
    * this example program, any retval which is not equal to NC_NOERR
    * (0) will cause the program to print an error message and exit
    * with a non-zero return code. */

   /* Create the file. The NC_CLOBBER parameter tells netCDF to
    * overwrite this file, if it already exists.*/
   if ((retval = nc_create(FILE_NAME, NC_CLOBBER, &ncid)))
      ERR(retval);

   /* Define the dimensions. NetCDF will hand back an ID for each. */
   if ((retval = nc_def_dim(ncid, "x", NX, &x_dimid)))
      ERR(retval);
   if ((retval = nc_def_dim(ncid, "y", NY, &y_dimid)))
      ERR(retval);

   /* The dimids array is used to pass the IDs of the dimensions of
    * the variable. */
   dimids[0] = x_dimid;
   dimids[1] = y_dimid;

   /* Define the variable. The type of the variable in this case is
    * NC_INT (4-byte integer). */
   if ((retval = nc_def_var(ncid, "data", NC_INT, NDIMS, 
			    dimids, &varid)))
      ERR(retval);

   /* End define mode. This tells netCDF we are done defining
    * metadata. */
   if ((retval = nc_enddef(ncid)))
      ERR(retval);

   /* Write the pretend data to the file. Although netCDF supports
    * reading and writing subsets of data, in this case we write all
    * the data in one operation. */
   if ((retval = nc_put_var_int(ncid, varid, &data_out[0][0])))
      ERR(retval);

   /* Close the file. This frees up any internal netCDF resources
    * associated with the file, and flushes any buffers. */
   if ((retval = nc_close(ncid)))
      ERR(retval);

   printf("*** SUCCESS writing example file simple_xy.nc!\n");
   return 0;
}

The code is compiled in a MSYS2 shell as follows:

$ gcc -o simple_xy_wr_nc481 -I/c/Users/Alexander\ Barth/.julia/artifacts/ae78b073115f5cca9ab13a23994bdd930ecfe887/include/ simple_xy_wr2.c -L/c/Users/Alexander\ Barth/.julia/artifacts/ae78b073115f5cca9ab13a23994bdd930ecfe887/lib -lnetcdf

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ ./simple_xy_wr_nc481.exe
Error: No such file or directory

The path /c/Users/Alexander Barth/.julia/artifacts/ae78b073115f5cca9ab13a23994bdd930ecfe887/ contains the NetCDF 4.8.1 library.

Prior to all tests, I checked that the file to be written does not exists (any more).

[DennisHeimbigner]

I suspect this is the same problem as this: #2105

I had an incorrect understanding of how mingw (msys actually) dealt with paths
inside its version of bash.

[Alexander-Barth]

It seems to be related, but here the error is independent of the tests.

[DennisHeimbigner]

Are you in a position to use this PR: https://github.com/Unidata/netcdf-c/pull/2105/files?
The PR proposes aliasing pwd. You may need to do the same in your shell
when running the example program.

[Alexander-Barth]

I tried to build NetCDF (./bootstrap; ./configure; make -j4) from the PR 2105, but I got this error:

/bin/sh ../libtool  --tag=CC   --mode=link gcc  -fno-strict-aliasing   -o ncgen3.exe main.o load.o escapes.o getfill.o init.o genlib.o ncgeny.o ../liblib/libnetcdf.la -lhdf5_hl -lhdf5 -lm -lz -lcurl
libtool: link: gcc -fno-strict-aliasing -o .libs/ncgen3.exe main.o load.o escapes.o getfill.o init.o genlib.o ncgeny.o  ../liblib/.libs/libnetcdf.a -lhdf5_hl -lhdf5 -lz -lcurl
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x12c5): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2901): undefined reference to `ncpsharedlibnew'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2924): undefined reference to `ncpload'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2944): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x295b): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2972): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2989): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2be5): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2c92): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2caf): undefined reference to `ncpsharedlibfree'
collect2.exe: error: ld returned 1 exit status
make[2]: *** [Makefile:739: ncgen3.exe] Error 1
make[2]: Leaving directory '/home/Alexander Barth/netcdf-c/ncgen3'
make[1]: *** [Makefile:738: all-recursive] Error 1
make[1]: Leaving directory '/home/Alexander Barth/netcdf-c'
make: *** [Makefile:583: all] Error 2

Alexander Barth@alex-laptop MINGW64 ~/netcdf-c
$

Alexander Barth@alex-laptop MINGW64 ~/netcdf-c
$ git log | head
commit 3749328fbd142600dff69814ca660d78baa5f7c7
Author: Milton Woods <miltonjwoods@gmail.com>
Date:   Sun Sep 5 19:30:48 2021 +1000

    Use Windows format for pwd in test_common on mingw

commit 2da66c641e296157fd4786a0ea2eef9eda58eb7c
Author: Milton Woods <miltonjwoods@gmail.com>
Date:   Sun Sep 5 19:37:08 2021 +1000

I got the same error with make -j4 LDFLAGS="-no-undefined".

[DennisHeimbigner]

You may have to apply the patches by hand rather than using the commit.

[Alexander-Barth]

I used these instructions to test the PR #2105 locally. I can confirm that I have these changes:

https://github.com/Unidata/netcdf-c/pull/2105/files

$ grep MSYS2 ncdump/testpathcvt.sh
export MSYS2_ARG_CONV_EXCL='*'

$ grep 'pwd -W' test_common.in
  alias pwd='pwd -W'

[DennisHeimbigner]

I just noticed something, can you change "ALEXAN~1" in that path to the actual full path?

[Alexander-Barth]

Unfortunately, I get still the same error (using the cross-compiled NetCDF library).

$ gcc -o simple_xy_wr_nc481 -I/c/Users/Alexander\ Barth/.julia/artifacts/ae78b073115f5cca9ab13a23994bdd930ecfe887/include/ simple_xy_wr3.c -L/c/Users/Alexander\ Barth/.julia/artifacts/ae78b073115f5cca9ab13a23994bdd930ecfe887/lib -lnetcdf


Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ ./simple_xy_wr_nc481.exe
Error: No such file or directory

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ diff simple_xy_wr3.c simple_xy_wr2.c
24c24
< #define FILE_NAME "C:\\Users\\Alexander Barth\\AppData\\Local\\Temp\\foobar.nc"
---
> #define FILE_NAME "C:\\Users\\ALEXAN~1\\AppData\\Local\\Temp\\foobar.nc"

[DennisHeimbigner]

I have no idea why this is failing. One more try; can you move the file to a location
so that the path has no blanks?

[Alexander-Barth]

With #define FILE_NAME "C:\\Test\\foobar.nc" it also fails, but it works with #define FILE_NAME "\\Test\\foobar.nc" (C: is my current drive, both file path correspond to the same location). So this issue seems to be related to the file drive parsing. As far as I can tell, there has been at lot of improvement related to Zarr in this release (which use a special file path), so maybe this is related.

[DennisHeimbigner]

If you use this path, does it work?

/c/Test/foobar.nc

[DennisHeimbigner]

I think the problem is my understanding of msys vs mingw.
My path converter checks for the MSYS pre-processor flag
to signal that it is under msys. For msys, it converts a windows path
c:\x\y to /c/x.y, which I understood to be the msys preferred path format.
I was assuming that mingw was the same as msys and so make no special
check for mingw.
Can you tell me the difference and what I should be doing in terms of path conversion?

[DennisHeimbigner]

Can you see if this patch fixes the problem:

1. edit netcdf-c/libdispatch/dpathmgr.c
2. Go to the function get_local_path_kind()
3. Change these lines:

#ifdef __CYGWIN__
	kind = NCPD_CYGWIN;

to these lines:

#ifdef __CYGWIN__
	kind = NCPD_CYGWIN;
#elif __MINGW32__
	kind = NCPD_WIN;

[Alexander-Barth]

When I try a native compilation (within the msys/mingw32 environement), I still get the error above:

mv -f $depbase.Tpo $depbase.Po
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -fno-strict-aliasing   -o ncgen3.exe main.o load.o escapes.o getfill.o init.o genlib.o ncgeny.o ../liblib/libnetcdf.la -lhdf5_hl -lhdf5 -lm -lz -lcurl
libtool: link: gcc -fno-strict-aliasing -o .libs/ncgen3.exe main.o load.o escapes.o getfill.o init.o genlib.o ncgeny.o  ../liblib/.libs/libnetcdf.a -lhdf5_hl -lhdf5 -lz -lcurl
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x12c5): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2901): undefined reference to `ncpsharedlibnew'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2924): undefined reference to `ncpload'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2944): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x295b): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2972): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2989): undefined reference to `ncpgetsymbol'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2be5): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2c92): undefined reference to `ncpsharedlibfree'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../liblib/.libs/libnetcdf.a(libnczarr_la-zfilter.o):zfilter.c:(.text+0x2caf): undefined reference to `ncpsharedlibfree'
collect2.exe: error: ld returned 1 exit status
make[2]: *** [Makefile:739: ncgen3.exe] Error 1
make[2]: Leaving directory '/home/Alexander Barth/netcdf-c/ncgen3'
make[1]: *** [Makefile:738: all-recursive] Error 1
make[1]: Leaving directory '/home/Alexander Barth/netcdf-c'
make: *** [Makefile:583: all] Error 2

With these changes (on top of #2105):

diff --git a/libdispatch/dpathmgr.c b/libdispatch/dpathmgr.c
index b8495be7..563762d4 100644
--- a/libdispatch/dpathmgr.c
+++ b/libdispatch/dpathmgr.c
@@ -862,6 +862,8 @@ getlocalpathkind(void)
     int kind = NCPD_UNKNOWN;
 #ifdef __CYGWIN__
        kind = NCPD_CYGWIN;
+#elif __MINGW32__
+       kind = NCPD_WIN;
 #elif __MSYS__
        kind = NCPD_MSYS;
 #elif _MSC_VER /* not _WIN32 */

Or should this patch be tried in cross-compilation?

I have to say, I am not so familiar with MSYS vs Mingw32. I am using Linux myself. I am just trying to make sure that my code runs all all major platforms :-)

I tried to use gcc -dM -E to see which macros are defined, and it seems that only _MINGW32_ is defined:

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ which gcc
/mingw64/bin/gcc

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ gcc -dM -E simple_xy_wr3.c | grep MSYS

Alexander Barth@alex-laptop MINGW64 /c/Users/Alexander Barth/Downloads
$ gcc -dM -E simple_xy_wr3.c | grep MINGW32
#define __MINGW32_MAJOR_VERSION 3
#define __MINGW32__ 1
#define __MINGW32_MINOR_VERSION 11

[Alexander-Barth]

It seems that the compiler errors are related to the fact that libncpoco is missing during linking.

Note that I use make LDFLAGS=-no-undefined. I am wondering if this can interfere with finding relevant libraries (more info).
Adding this library by hand (and disabling utilities) allows to go furter,

 /bin/sh ../libtool  --tag=CC   --mode=link gcc  -fno-strict-aliasing -version-info 19:1:0 -no-undefined -o libnetcdf.la -rpath /mingw64/lib libnetcdf_la-nc_initialize.lo ../libdispatch/libnetcdf2.la ../libdispatch/libdispatch.la ../libsrc/libnetcdf3.la  ../libhdf5/libnchdf5.la  ../libdap2/libdap2.la ../oc2/liboc.la ../libdap4/libdap4.la ../libsrc4/libnetcdf4.la ../libnczarr/libnczarr.la  ../libncpoco/libncpoco.la   -lhdf5_hl -lhdf5 -lm -lz -lcurl -ldl

but then I have this error:

make[3]: Entering directory '/home/Alexander Barth/netcdf-c/plugins'
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -fno-strict-aliasing  -module -avoid-version -shared -export-dynamic -rpath /home/Alexander Barth/netcdf-c/plugins -no-undefined -no-undefined -o libnczdefaults.la  NCZdefaults.lo  -lhdf5_hl -lhdf5 -lm -lz -lcurl
libtool: link: rm -fr  .libs/libnczdefaults.dll.a
libtool: link: gcc -shared  .libs/NCZdefaults.o   -lhdf5_hl -lhdf5 -lz -lcurl    -o .libs/libnczdefaults.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libnczdefaults.dll.a
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x18c3): undefined reference to `nc_inq_var'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x18f9): undefined reference to `nc_inq_type'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x21ce): undefined reference to `nc_inq_var'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x2204): undefined reference to `nc_inq_type'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x229d): undefined reference to `nc_inq_dimlen'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x22d4): undefined reference to `nc_inq_var_chunking'
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: .libs/NCZdefaults.o:NCZdefaults.c:(.text+0x2368): undefined reference to `nc_inq_var_endian'
collect2.exe: error: ld returned 1 exit status

[DennisHeimbigner]

The libncpoco was added after 4.8.1 was released. Your best bet might be to go back to
a pristine 4.8.1 and apply the change above.

[Alexander-Barth]

OK, I understand. On 4.8.1 I applied these changes (the file ncdump/testpathcvt.sh from https://github.com/Unidata/netcdf-c/pull/2105/files does not seem to exist in this release):

diff --git a/libdispatch/dpathmgr.c b/libdispatch/dpathmgr.c
index eb450534..a788d74a 100644
--- a/libdispatch/dpathmgr.c
+++ b/libdispatch/dpathmgr.c
@@ -859,6 +859,8 @@ getlocalpathkind(void)
     int kind = NCPD_UNKNOWN;
 #ifdef __CYGWIN__
        kind = NCPD_CYGWIN;
+#elif __MINGW32__
+        kind = NCPD_WIN;
 #elif __MSYS__
        kind = NCPD_MSYS;
 #elif _MSC_VER /* not _WIN32 */
diff --git a/test_common.in b/test_common.in
index 5e36b301..2e09767c 100644
--- a/test_common.in
+++ b/test_common.in
@@ -73,6 +73,14 @@ set -e
 # Allow global set -x mechanism for debugging.
 if test "x$SETX" = x1 ; then set -x ; fi

+# On MINGW, bash and other POSIX utilities use a mounted root directory,
+# but executables compiled for Windows do not recognise the mount point.
+# Here we ensure that Windows paths are used in tests of Windows executables.
+system=`uname`
+if test "x${system##MINGW*}" = x; then
+  alias pwd='pwd -W'
+fi
+
 # We assume that TOPSRCDIR and TOPBUILDDIR are defined
 # At the top of this shell script
 top_srcdir="$TOPSRCDIR"

I compiled with:

make -j4 LDFLAGS="-no-undefined -Wl,--export-all-symbols"

(Note the -Wl,--export-all-symbols is now necessary)

Compilation succeeds! (horray!! :-) ).

When I go back at my initial test programs, now all paths I tried do work! I tried these:

/*#define FILE_NAME "C:\\Test\\foobar.nc" /* works */
/*#define FILE_NAME "/c/Test/foobar.nc" /* works */
/*#define FILE_NAME "C:\\Users\\ALEXAN~1\\AppData\\Local\\Temp\\foobar.nc" /* works */
#define FILE_NAME "C:\\Users\\Alexander Barth\\AppData\\Local\\Temp\\foobar.nc" /* works */

However, there are still some issues related to the test suite where I see some failures:

============================================
   netCDF 4.8.1: unit_test/test-suite.log
============================================

# TOTAL: 6
# PASS:  5
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_xcache
================

Assertion failed: rnd > ' ' && rnd < 127, file tst_xcache.c, line 97
FAIL tst_xcache.exe (exit status: 3)

The relevant code:

        for(k=0;k<len;k++) {
            do {rnd = random() % 127;} while(rnd < ' ');
            assert(rnd > ' ' && rnd < 127);
            s[k] = (char)rnd;
        }

Should it be the following to exclude the space character?

diff --git a/unit_test/tst_xcache.c b/unit_test/tst_xcache.c
index 0c61dd83..d527fcd5 100644
--- a/unit_test/tst_xcache.c
+++ b/unit_test/tst_xcache.c
@@ -93,7 +93,7 @@ generatestrings(int n, unsigned seed)
        len = rnd % MAXSTRLEN;
        /* generate the characters */
        for(k=0;k<len;k++) {
-           do {rnd = random() % 127;} while(rnd < ' ');
+           do {rnd = random() % 127;} while(rnd <= ' ');
            assert(rnd > ' ' && rnd < 127);
            s[k] = (char)rnd;
        }

After this change, I see this issue in tst_diskless6:

==========================================
   netCDF 4.8.1: nc_test/test-suite.log
==========================================

# TOTAL: 22
# PASS:  21
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_diskless6
===================

*** Test create
test: file=file3d.nc mode=NC_NETCDF3|NC_DISKLESS
test: file=file3dp.nc mode=NC_NETCDF3|NC_DISKLESS|NC_PERSIST
test: file=file4d.nc mode=NC_NETCDF4|NC_DISKLESS
test: file=file4dp.nc mode=NC_NETCDF4|NC_DISKLESS|NC_PERSIST
*** Test open + modify + rdwrite
test: file=file3dp.nc mode=NC_NETCDF3|NC_DISKLESS|NC_WRITE|NC_PERSIST
*** Fail: line: 223: (-68) NetCDF: I/O failure
FAIL tst_diskless6.exe (exit status: 1)

Thanks a lot for your help!

[DennisHeimbigner]

Good catch on the < ' '; I will add it to my PR.
I will have to check about the diskless6 problem.

[DennisHeimbigner]

For a start, edit tst_diskless6.c and change #define CLEANUP to #undef CLEANUP.
Then do make check again.
There should be a file named file3dp.nc.
Do an ncdump on that file.

[Alexander-Barth]

ncdump does not like this file (using ncdump 4.7.3 on Ubuntu 20.04):

$ ncdump  file3dp.nc
ncdump: file3dp.nc: file3dp.nc: Invalid argument

The file is actually quite small. I made it available here:
https://dox.ulg.ac.be/index.php/s/sN9iXArMiFuIs3d/download

[DennisHeimbigner]

It is a small file containing only the definitions of a couple of dimensions.
However it should be readable.

[DennisHeimbigner]

Took al look at it and it is short.

[DennisHeimbigner]

Try this experiment.

1. download attached file
2. rename it to memio.c
3. replace netcdf-c/libsrc/memio.c with the renamed file
4. rebuild
5. make test

memio.txt

[Alexander-Barth]

Thanks a lot! tst_diskless6.exe passes now:

[...]
PASS: tst_max_var_dims.exe
PASS: tst_formats.exe
PASS: tst_def_var_fill.exe
PASS: tst_err_enddef.exe
PASS: tst_default_format.exe
PASS: tst_diskless6.exe
PASS: run_cdf5.sh
[...]

Unfortunately, I have now these errors (later in the test suite):

PASS: test_parse.sh
FAIL: test_meta.sh
FAIL: test_data.sh
FAIL: test_fillmismatch.sh
FAIL: test_raw.sh
============================================================================
Testsuite summary for netCDF 4.8.1
============================================================================
# TOTAL: 5
# PASS:  1
# SKIP:  0
# XFAIL: 0
# FAIL:  4
# XPASS: 0
# ERROR: 0
============================================================================
See dap4_test/test-suite.log
Please report to support-netcdf@unidata.ucar.edu
============================================================================

Attached is the corresponding test-suite.log file.

test-suite.log

They seem to be related to the fact, that I disabled utilities (ncdump and friends). I will enable ncdump and report back.
Thanks again!

[Alexander-Barth]

Here are some "failed" test concerning ncdump:

PASS: tst_inttags.sh
PASS: run_tests.sh
PASS: tst_64bit.sh
PASS: ref_ctest.exe
PASS: ref_ctest64.exe
FAIL: tst_output.sh
PASS: tst_lengths.sh
PASS: tst_calendars.sh
PASS: run_utf8_tests.sh
FAIL: tst_nccopy3.sh
PASS: tst_nccopy3_subset.sh
PASS: tst_charfill.sh
PASS: tst_iter.sh
PASS: tst_formatx3.sh
PASS: tst_bom.sh
PASS: tst_dimsizes.sh
PASS: run_ncgen_tests.sh
PASS: tst_ncgen4_classic.sh
PASS: test_radix.sh
PASS: test_rcmerge.sh
PASS: tst_null_byte_padding.sh
FAIL: test_unicode_directory.sh
FAIL: test_unicode_path.sh
PASS: tst_fileinfo.sh
PASS: tst_hdf5_offset.sh
PASS: tst_inttags4.sh
PASS: tst_netcdf4.sh
PASS: tst_fillbug.sh
FAIL: tst_netcdf4_4.sh
PASS: tst_ncgen4.sh
PASS: tst_h_scalar.sh
PASS: tst_nccopy4.sh
PASS: tst_nccopy5.sh
PASS: tst_grp_spec.sh
PASS: tst_mud.sh
PASS: tst_formatx4.sh
PASS: run_utf8_nc4_tests.sh
FAIL: run_back_comp_tests.sh
PASS: run_ncgen_nc4_tests.sh
PASS: test_scope.sh
PASS: tst_inmemory_nc3.sh
PASS: tst_nccopy_w3.sh
PASS: tst_inmemory_nc4.sh
PASS: tst_nccopy_w4.sh
PASS: tst_ctests.sh
PASS: test_keywords.sh
============================================================================
Testsuite summary for netCDF 4.8.1
============================================================================
# TOTAL: 46
# PASS:  40
# SKIP:  0
# XFAIL: 0
# FAIL:  6
# XPASS: 0
# ERROR: 0

I am wondering if the test case is a bit too strict, as I see these differences:

209c209
<  f1 = -3.402823e+38 ;
---
>  f1 = -3.402823e+038 ;

test-suite.log

[DennisHeimbigner]

I can fix the precision problem. But fixing the e+038 vs e+38 problem is trickier.

[DennisHeimbigner]

IN looking at the build file, I see some of these are disabled if running under windows.
I expect that they should be disabled under mingw as well.
Can you do the "uname" command under mingw and send me the result?

[Alexander-Barth]

Short answer MINGW64_NT-10.0-19041 in the shell that I used so far.

Long answer: MSYS2 seems to come with 3 different launchers: MINGW32, MINGW64, MSYS2 (i.e. shells in a terminal with some environment variable setup; https://www.davidegrayson.com/windev/msys2/, https://www.msys2.org/wiki/Launchers/). They have all different unames.

launcher	uname
MINGW32	MINGW32_NT-10.0-19041
MINGW64	MINGW64_NT-10.0-19041
MSYS	MSYS_NT-10.0-19041

Sorry if this is confusing!

[DennisHeimbigner]

Fixed by #2138

[Alexander-Barth]

Thank you for this PR! I applied it manually on the version 4.8.1, but I get the error:

/bin/sh ../libtool  --tag=CC   --mode=compile gcc -DHAVE_CONFIG_H -I. -I..  -I../include -I../include -I../oc2  -I../libhdf5  -I../libdap2 -I../oc -I../libdap4 -I../libsrc4 -I../libnczarr   -fno-strict-aliasing -MT libnetcdf_la-nc_initialize.lo -MD -MP -MF .deps/libnetcdf_la-nc_initialize.Tpo -c -o libnetcdf_la-nc_initialize.lo `test -f 'nc_initialize.c' || echo './'`nc_initialize.c
libtool: compile:  gcc -DHAVE_CONFIG_H -I. -I.. -I../include -I../include -I../oc2 -I../libhdf5 -I../libdap2 -I../oc -I../libdap4 -I../libsrc4 -I../libnczarr -fno-strict-aliasing -MT libnetcdf_la-nc_initialize.lo -MD -MP -MF .deps/libnetcdf_la-nc_initialize.Tpo -c nc_initialize.c  -DDLL_EXPORT -DPIC -o .libs/libnetcdf_la-nc_initialize.o
mv -f .deps/libnetcdf_la-nc_initialize.Tpo .deps/libnetcdf_la-nc_initialize.Plo
/bin/sh ../libtool  --tag=CC   --mode=link gcc  -fno-strict-aliasing -version-info 19:1:0 -no-undefined  -Wl,--export-all-symbols -o libnetcdf.la -rpath /mingw64/lib libnetcdf_la-nc_initialize.lo ../libdispatch/libnetcdf2.la ../libdispatch/libdispatch.la ../libsrc/libnetcdf3.la  ../libhdf5/libnchdf5.la  ../libdap2/libdap2.la ../oc2/liboc.la ../libdap4/libdap4.la ../libsrc4/libnetcdf4.la ../libnczarr/libnczarr.la  -lhdf5_hl -lhdf5 -lm -lz -lcurl
libtool: link: gcc -shared  .libs/libnetcdf_la-nc_initialize.o  -Wl,--whole-archive ../libdispatch/.libs/libnetcdf2.a ../libdispatch/.libs/libdispatch.a ../libsrc/.libs/libnetcdf3.a ../libhdf5/.libs/libnchdf5.a ../libdap2/.libs/libdap2.a ../oc2/.libs/liboc.a ../libdap4/.libs/libdap4.a ../libsrc4/.libs/libnetcdf4.a ../libnczarr/.libs/libnczarr.a -Wl,--no-whole-archive  -lhdf5_hl -lhdf5 -lz -lcurl  -Wl,--export-all-symbols   -o .libs/libnetcdf-19.dll -Wl,--enable-auto-image-base -Xlinker --out-implib -Xlinker .libs/libnetcdf.dll.a
C:/msys64/mingw64/bin/../lib/gcc/x86_64-w64-mingw32/10.1.0/../../../../x86_64-w64-mingw32/bin/ld.exe: ../libdispatch/.libs/libdispatch.a(libdispatch_la-drc.o):drc.c:(.text+0x103): undefined reference to `assert'
collect2.exe: error: ld returned 1 exit status
make[2]: *** [Makefile:581: libnetcdf.la] Error 1
make[2]: Leaving directory '/home/Alex/netcdf-c/liblib'
make[1]: *** [Makefile:729: all-recursive] Error 1
make[1]: Leaving directory '/home/Alex/netcdf-c'
make: *** [Makefile:574: all] Error 2

But adding, #include <assert.h> in the file libdispatch/drc.c solves this compilation issue.

For some strange reasons, the test tst_output.sh still gets called (and fails), despite I have this change:

diff --git a/ncdump/Makefile.am b/ncdump/Makefile.am
index 8b527aa6..244cc7a3 100644
--- a/ncdump/Makefile.am
+++ b/ncdump/Makefile.am
@@ -72,8 +72,8 @@ bom tst_dimsizes nctrunc tst_rcmerge

 # Tests for classic and 64-bit offset files.
 TESTS = tst_inttags.sh run_tests.sh tst_64bit.sh ref_ctest     \
-ref_ctest64 tst_output.sh tst_lengths.sh tst_calendars.sh      \
-run_utf8_tests.sh tst_nccopy3.sh tst_nccopy3_subset.sh         \
+ref_ctest64 tst_lengths.sh tst_calendars.sh    \
+run_utf8_tests.sh tst_nccopy3_subset.sh                \
 tst_charfill.sh tst_iter.sh tst_formatx3.sh tst_bom.sh         \
 tst_dimsizes.sh run_ncgen_tests.sh tst_ncgen4_classic.sh test_radix.sh test_rcmerge.sh

@@ -87,10 +87,6 @@ if USE_STRICT_NULL_BYTE_HEADER_PADDING
 XFAIL_TESTS += tst_null_byte_padding.sh
 endif

-if ! ISCYGWIN
-TESTS += test_unicode_directory.sh  test_unicode_path.sh
-endif
-
 if LARGE_FILE_TESTS
 TESTS += tst_iter.sh
 endif
@@ -107,9 +103,9 @@ check_PROGRAMS += tst_vlen_demo

 # Tests for netCDF-4 behavior.
 TESTS += tst_fileinfo.sh tst_hdf5_offset.sh tst_inttags4.sh            \
-tst_netcdf4.sh tst_fillbug.sh tst_netcdf4_4.sh tst_nccopy4.sh          \
+tst_netcdf4.sh tst_fillbug.sh tst_nccopy4.sh           \
 tst_nccopy5.sh tst_grp_spec.sh tst_mud.sh tst_h_scalar.sh tst_formatx4.sh              \
-run_utf8_nc4_tests.sh run_back_comp_tests.sh run_ncgen_nc4_tests.sh    \
+run_utf8_nc4_tests.sh run_ncgen_nc4_tests.sh   \
 tst_ncgen4.sh test_scope.sh

 # Record interscript dependencies so parallel builds work.
@@ -133,6 +129,16 @@ if ENABLE_CDF5
 TESTS += test_keywords.sh
 endif

+if ! ISMINGW
+if ! ISCYGWIN
+TESTS += tst_output.sh tst_nccopy3.sh
+TESTS += test_unicode_directory.sh  test_unicode_path.sh
+if USE_HDF5
+TESTS += run_back_comp_tests.sh tst_netcdf4_4.sh
+endif
+endif
+endif
+
 endif BUILD_TESTSETS

 # These files all have to be included with the distribution.
@@ -220,4 +226,5 @@ scope_*.nc copy_scope_*.cdl

 # Remove directories
 clean-local:
-       rm -fr rcmergedir
+       rm -fr rcmergedir rchome
+

I still see the test in ncdump/Makefile (after running ./configure):

TESTS = tst_inttags.sh run_tests.sh tst_64bit.sh \
        ref_ctest$(EXEEXT) ref_ctest64$(EXEEXT) \
        tst_output.sh tst_lengths.sh \
        tst_calendars.sh run_utf8_tests.sh \
        tst_nccopy3.sh tst_nccopy3_subset.sh \
        tst_charfill.sh tst_iter.sh \
        tst_formatx3.sh tst_bom.sh \
        tst_dimsizes.sh run_ncgen_tests.sh \
        tst_ncgen4_classic.sh test_radix.sh \
        test_rcmerge.sh tst_null_byte_padding.sh \
        $(am__append_9) $(am__append_10) \
        $(am__append_12) tst_inmemory_nc3.sh \
        tst_nccopy_w3.sh $(am__append_13) \
        $(am__append_14)

[DennisHeimbigner]

Odd, drc.c appears to already contain #include <assert.h>.

The other problem is that this Makefile.am test is not working right.

if ! ISMINGW
if ! ISCYGWIN

[DennisHeimbigner]

Try changing to the following in ncdump/Makefile.am.

if !ISMINGW
if !ISCYGWIN

and see if that fixes the problem,

[DennisHeimbigner]

fixed

[Alexander-Barth]

With this change (and rerunning ./bootstrap, I am not sure if this was necessary), I can confirm that the test tst_output.sh is now skipped. I still see now an issue in tst_nccopy4.sh.

I am wondering if it would not be better to have a CI system testing mingw64. This would speed up significantly the testing and for you it would be much clearer to see what change did results in which outcome. I would be glad to help to set this up.

PASS: tst_ncgen4.sh
PASS: tst_h_scalar.sh
FAIL: tst_nccopy4.sh
PASS: tst_nccopy5.sh
PASS: tst_grp_spec.sh
PASS: tst_mud.sh
PASS: tst_formatx4.sh
PASS: run_utf8_nc4_tests.sh
PASS: run_ncgen_nc4_tests.sh
PASS: test_scope.sh
PASS: tst_inmemory_nc3.sh
PASS: tst_nccopy_w3.sh
PASS: tst_inmemory_nc4.sh
PASS: tst_nccopy_w4.sh
PASS: tst_ctests.sh
PASS: test_keywords.sh
============================================================================
Testsuite summary for netCDF 4.8.1
============================================================================
# TOTAL: 40
# PASS:  39
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0
============================================================================
See ncdump/test-suite.log
Please report to support-netcdf@unidata.ucar.edu
============================================================================
make[4]: *** [Makefile:1518: test-suite.log] Error 1
make[4]: Leaving directory '/home/Alex/netcdf-c/ncdump'
make[3]: *** [Makefile:1626: check-TESTS] Error 2
make[3]: Leaving directory '/home/Alex/netcdf-c/ncdump'
make[2]: *** [Makefile:1732: check-am] Error 2
make[2]: Leaving directory '/home/Alex/netcdf-c/ncdump'
make[1]: *** [Makefile:1405: check-recursive] Error 1
make[1]: Leaving directory '/home/Alex/netcdf-c/ncdump'
make: *** [Makefile:731: check-recursive] Error 1


Alexander Barth@alex-laptop MINGW64 /home/Alex/netcdf-c
$ cd ncdump/

Alexander Barth@alex-laptop MINGW64 /home/Alex/netcdf-c/ncdump
$ cat test-suite.log
=========================================
   netCDF 4.8.1: ncdump/test-suite.log
=========================================

# TOTAL: 40
# PASS:  39
# SKIP:  0
# XFAIL: 0
# FAIL:  1
# XPASS: 0
# ERROR: 0

.. contents:: :depth: 2

FAIL: tst_nccopy4
=================


*** Testing compound types some more.
*** creating another compound test file tst_comp2.nc...ok.
*** Tests successful!

*** Testing netCDF-4 features of nccopy on ncdump/*.nc files
*** Test nccopy tst_comp.nc copy_of_tst_comp.nc ...
*** compare  with copy_of_tst_comp.cdl
*** Test nccopy tst_comp2.nc copy_of_tst_comp2.nc ...
*** compare  with copy_of_tst_comp2.cdl
*** Test nccopy tst_enum_data.nc copy_of_tst_enum_data.nc ...
*** compare  with copy_of_tst_enum_data.cdl
*** Test nccopy tst_fillbug.nc copy_of_tst_fillbug.nc ...
*** compare  with copy_of_tst_fillbug.cdl
*** Test nccopy tst_group_data.nc copy_of_tst_group_data.nc ...
*** compare  with copy_of_tst_group_data.cdl
*** Test nccopy tst_nans.nc copy_of_tst_nans.nc ...
*** compare  with copy_of_tst_nans.cdl
*** Test nccopy tst_opaque_data.nc copy_of_tst_opaque_data.nc ...
*** compare  with copy_of_tst_opaque_data.cdl
*** Test nccopy tst_solar_1.nc copy_of_tst_solar_1.nc ...
*** compare  with copy_of_tst_solar_1.cdl
*** Test nccopy tst_solar_2.nc copy_of_tst_solar_2.nc ...
*** compare  with copy_of_tst_solar_2.cdl
*** Test nccopy tst_solar_cmp.nc copy_of_tst_solar_cmp.nc ...
*** compare  with copy_of_tst_solar_cmp.cdl
*** Test nccopy tst_special_atts.nc copy_of_tst_special_atts.nc ...
*** compare  with copy_of_tst_special_atts.cdl
*** Test nccopy tst_string_data.nc copy_of_tst_string_data.nc ...
No such file or directory
Location: file nccopy.c; line 2018
FAIL tst_nccopy4.sh (exit status: 1)

[DennisHeimbigner]

Is it possible to set up a github action to test mingw?

[dopplershift]

@DennisHeimbigner Somebody just needs to codify the steps it takes to setup the environment and build netcdf for Windows. But yes, GitHub actions has support for Windows, Linux (ubuntu), and macOS.

[DennisHeimbigner]

Ward- have any idea how to approach this?

[WardF]

@dopplershift I'll look, it may be a little stale but these directions were written and distributed when cmake support was originally introduced. The primary issue (at the time, based on the feedback we received) was the overhead of setting up Visual Studio + compiling the required libraries by hand was a big lift for users who wanted to just use the libraries, not necessarily develop against them. I'll take a look at refreshing this and see what libraries still need to be compiled by hand vs. which ones can be linked against or installed by a package management system like conda, although if they are using conda to install dependencies perhaps they should just use it to install libnetcdf altogether.

@DennisHeimbigner Yes, I'll take a look at the Github Actions platform environments. Saying it has support for 'Windows' is a bit nebulous, insofar as sometimes people say Windows when they mean Visual Studio (which is the habit I'm trying to break), and others use it to mean literally any of the popular development environments (MSYS+MinGW, CMD+MinGW, Powershell+MinGW, Cygwin) existing within a Windows OS environment.

In either case, we need to automated more Windows platform testing, regardless of the dev. environment, and perhaps that will be the next priority after the current writing project is in a good place.

[dopplershift]

Well in this case GitHub actions supporting Windows means you can start up a VM, that happens to be pre-populated with a bunch of dev packages. You can of course install on top of that pretty much whatever you feel like.

[DennisHeimbigner]

In this case all we need is a windows machine running mingw. I don't think that requires
using visual studio at all.

[WardF]

@DennisHeimbigner That is correct. I will take a look at the Github Actions documentation and see what I can set up.

[Alexander-Barth]

Here is an attempt from my side:
https://github.com/Alexander-Barth/NetCDF-CI-Builder/blob/main/.github/workflows/ci.yml
https://github.com/Alexander-Barth/NetCDF-CI-Builder/blob/main/ci-build.sh

Which is directly based on this:

https://www.msys2.org/docs/ci/

Msys2 provides a github action which make this quite easy to do.

[DennisHeimbigner]

Ok, I have a working mys/mingw action workflow.
It is disclosing a number of issues that need fixing.

[DennisHeimbigner]

I am tearing my hair out over this. I am beginning to think that msys2 path handling
is so screwed up it is unusable.

[Alexander-Barth]

In julia we are still using NetCDF 4.7.4 but there are some issues (JuliaGeo/NCDatasets.jl#162, JuliaGeo/NCDatasets.jl#145) in this version which are solved in NetCDF 4.8. The julia package NetCDF.jl from @meggart uses the same NetCDF compiled library and also currently at 4.7.4.

It would be great if we could update to a newer version (also for Zarr support). Unfortunately, Windows/mingw prevents us from upgrading. Have some of the changes mentioned in this thread already being integrated in the main branch?

[Alexander-Barth]

As far as I know, this is fixed by DennisHeimbigner@55a2643
Thanks a lot!