NetCDF scale factor not being processed

[malloryprow]

I've been working with NESDIS to get their ALEXI/GET-D product NetCDF output in a format that MET can process. We have made some great progress, but I'm not sure if the scale factor is being processed by MET.

I ran the file through plot_data_plane to plot HDAY (or daily sensible heat flux) which shows plane min: -47204, plane max: 185084.

If I use ncivew to view the variable the displayed range is -4.31 to 16.7494 MJ/m2/day.

I'm wondering if the file isn't set up right to get MET to process the scale factor correctly? I was wondering if someone could take a look? I tried to add a .zip of the data file, output from plot_data_plane, and the GrADS images from NESDIS, but the file was too big!

[barlage]

@malloryprow and others: There is a sample of the data on Cheyenne (/glade/scratch/barlage/getd/sample_data) if that's easy/easier for the MET crew to access.

[j-opatz]

Hi Mallory,

And thanks for your question. I'm going to take a look at this data today (thanks to @barlage for pointing out where to find a sample of this!) to see what's going on, but my first guess is that MET is processing the data correctly: ncview takes a subsample of the full data to plot and oftentimes misses the finer details of the entire dataset, including the full plane scale.

As I mentioned, I'll doublecheck this and see what I can find.

A question for you: what should be the proper range for this data? Is MET correct (-47204 to 185084), or should it be the smaller scale that ncview displays (-4.31 to 16.7494)? This could also be a scale issue that can be corrected via the convert entry.

[malloryprow]

Thanks for looking into this! It should be the smaller scale values we are seeing with ncview.

[barlage]

@j-opatz as you'll see in the sample data, the scale_factor is 0.0001 (1/10000), so it seems that MET is not applying the scale_factor to the data; MET is reporting the stored integer values. Also note that the numbers being mentioned here are for a different date than what I had on glade, but they are ~correct.

[j-opatz]

Thanks to both,

I was able to get in and see what there was to see. This is definitely a scale issue: I've attached two images from plot_data_plane.
The first image (Noscale.png) shows the data as MET sees it without any corrections via the call

plot_data_plane /glade/scratch/barlage/getd/sample_data/GETDL3_DAL_CONUS_2021182_1.0.nc testing_noscale.ps 'name="HDAY"; level="(*,*)";'

The second image (scaled_image.png) shows what should occur with the scaling factor applied. I was able to accomplish this with

plot_data_plane /glade/scratch/barlage/getd/sample_data/GETDL3_DAL_CONUS_2021182_1.0.nc testing_scale.ps 'name="HDAY"; level="(*,*)"; convert(x)=x*0.0001;'

I'm going to tag @JohnHalleyGotway for a reference here, as I don't know if MET has (or should have) the capability to natively decode a scale_factor tag for netCDF data. From the above we can see that MET's convert option makes quick work of data that's not to the proper scale, but should that be taken care of automatically in data that has a scale_factor?

[JohnHalleyGotway]

Hi all, @j-opatz asked me to take a closer look at this. I suspect that this is the first time we've run across NetCDF files that specify the scale_factor but not add_offset. And that's why we're seeing this behavior.

On this line, the code is only parsing scale_factor and add_offset if BOTH are present. We could easily tweak that logic to process them independently.

I read a bit in the CF-conventions doc, and saw this comment:

If both scale_factor and add_offset attributes are present, the data are first scaled before the offset is added.

Based on that, I infer that neither, one, or both can be present, and they should be processed independently. I'll write up a bugfix issue about this.

In the meantime, two options I see are...

1. Add the add_offset = 0 ; variable attribute to trigger the existing logic.
2. Specify the convert(x) = x*0.0001; config option that @j-opatz described above.

[barlage]

Thanks @JohnHalleyGotway. We have an additional request for the data developer so we can also request the add_offset to be added. It's probably good practice to include both.

[JohnHalleyGotway]

OK, here's the bugfix issue for this: dtcenter/MET#2187

I'll fix it in both the main_v10.1 branch (for a potential 10.1.3 bugfix release) and the develop branch (for the next 11.0.0 official release). Should be an easy fix. The harder part is getting an updated release compiled where you need it.