_FillValue attribute erases previous data

[Dave-Allured]

Platform: Mac OS X, 64 bit
Build system: Via Macports.org, uses autotools I think.
netcdf version: 4.4.1

Summary of Issue:

Writing a new _FillValue attribute to a netcdf-4 file variable with previous data should give an error. Instead, the _FillValue is written, and previous data is erased.

Steps to reproduce:

1. Start with an existing netcdf-4 file, and a variable with previous data, but NO DEFINED _FILLVALUE.
2. Open file in read/write mode.
3. Enter redefine mode (required with netcdf-4 classic format, optional with plain netcdf-4).
4. Call nc_put_att, define a new _FillValue. Error expected here.
5. Call nc_close.
6. Compare netcdf files, before and after.

Here is a sample program to write the new _FillValue attribute, thanks to @wkliao whose program from related issue #384 I modified:

#include <stdio.h>
#include <netcdf.h>

#define ERR {if(err!=NC_NOERR)printf("Error at line %d: %s\n",__LINE__,nc_strerror(err));}
int
main (int argc, char **argv)
{
    int ncid, varid, fmode, err;
    int fillv;

    fmode = NC_WRITE;
    err = nc_open ("testfile.nc", fmode, &ncid); ERR
    err = nc_inq_varid (ncid, "var", &varid); ERR

    err = nc_redef (ncid); ERR

    /* try put attribute _FillValue and expect NC_ELATEFILL */
    fillv = 99;
    err = nc_put_att_int (ncid, varid, _FillValue, NC_INT, 1, &fillv);
    if (err != NC_ELATEFILL)
        printf("line %d expecting NC_ELATEFILL but got %d\n",__LINE__,err);
    err = nc_close (ncid); ERR
    return 0;
}

I expect nc_put_att_int to report NC_ELATEFILL and leave the netcdf file untouched. Instead it reports unexpected NC_NOERR (0), and the netcdf file is changed:

3> gcc -g put-att1.c -I/opt/local/include -L/opt/local/lib -lnetcdf
4> ls -go testfile.nc 
-rw-r--r--  1   13542 Mar 28 18:45 testfile.nc
5> ./a.out
line 21 expecting NC_ELATEFILL but got 0
6> ls -go testfile.nc
-rw-r--r--  1   13810 Mar 28 19:28 testfile.nc

Sample netcdf file BEFORE modification. You can make your own test file by running this through ncgen:

netcdf testfile {
dimensions:
	dim = 10 ;
variables:
	int var(dim) ;
		var:_Storage = "contiguous" ;
		var:_Endianness = "little" ;

// global attributes:
		:_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17" ;
		:_SuperblockVersion = 0 ;
		:_IsNetcdf4 = 1 ;
		:_Format = "netCDF-4" ;
data:

 var = 11, 22, 33, 44, 55, _, _, _, _, _ ;
}

Test file AFTER running the put_var_att program. Note that five previous values at the start of "var" are erased, i.e. set to _FillValue:

netcdf testfile {
dimensions:
	dim = 10 ;
variables:
	int var(dim) ;
		var:_FillValue = 99 ;
		var:_Storage = "contiguous" ;
		var:_Endianness = "little" ;

// global attributes:
		:_NCProperties = "version=1|netcdflibversion=4.4.1|hdf5libversion=1.8.17" ;
		:_SuperblockVersion = 0 ;
		:_IsNetcdf4 = 1 ;
		:_Format = "netCDF-4" ;
data:

 var = _, _, _, _, _, _, _, _, _, _ ;
}

[DennisHeimbigner]

Why should this give an error?

[Dave-Allured]

nc_put_att documentation, possible error return:

NC_ELATEFILL -- Fill values must be written while the file is still in initial define mode.

HDF5 FAQ:

"the fill value for a dataset cannot be modified once the dataset has been created"
https://support.hdfgroup.org/HDF5/hdf5-quest.html#pfillvalue

[DennisHeimbigner]

I see I mis-interpreted ELATEFILL.
;

[wkliao]

Wow! I did not expect the erasure of previous written values (the first 5 elements).

[WardF]

Good to know, and this clarifies the speculation on the previous patch. It appears that we do indeed want to interpret this as the initial define mode.

[DennisHeimbigner]

Does this issue relate to pr's 383, 387, and 389?

[WardF]

I believe that it does; I'm reviewing those now that we have this additional info, and am incorporating the tests that @wkliao has provided.

[Dave-Allured]

You can mark this issue as low priority, as far as I am concerned. I stumbled on this behavior while I was curious about related issue #384. It is not a problem for my own applications.