P2G V3 HDF brightness temperatures are different from P2G V2.3

[kathys]

I noticed that the P2G V3 Brightness temperatures are different at the warm end of the calculations when compared with those created from P2G V2.3. They are significantly cooler at the warm end of the terrestrial temperatures. The values at the cold end look very, very similar.

I am attaching images that I created using Panoply for VIIRS I04 over a 1000x1000 300m grid that over Mexico.

grid: campo_dos_campo_mx, proj4, +proj=lcc +datum=WGS84 +ellps=WGS84 +lat_0=24.00000 +lat_1=24.00000 +lon_0=-103.39000 +units=m +no_defs, 1000, 1000, 300.00000, -300.00000, -104.87971deg, 25.34690deg

P2G v3.0 execution:
$POLAR2GRID_HOME/bin/polar2grid.sh -r viirs_sdr -w hdf5 -p i01 i02 i03 i04 i05 dynamic_dnb m06 m09 m10 m15 m16 --num-workers 4 --grid-configs /data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/test_grids.conf -g campo_dos_campo_mx -f /data/test_data/old_polar2grid_data/viirs_sdr_2021/input

P2G v2.3 exection:
$POLAR2GRID_HOME/bin/polar2grid.sh viirs_sdr hdf5 --p i01 i02 i03 i04 i05 dynamic_dnb m06 m09 m10 m15 m16 --grid-configs /data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/test_grids.conf -g campo_dos_campo_mx -f /data/test_data/old_polar2grid_data/viirs_sdr_2021/input

bumi output including logs:
P2G v3.0: /data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/new_p2g_hdf
P2G v2.3: /data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/orig_p2g_hdf

[djhoese]

Would it be possible to make the same screenshots with the same limits in the colorbar? Next question is, can you make something similar but with a larger section? I'm wondering (hoping 🤞) that since the hottest pixels are on the left of the image that maybe some small differences in remapping and/or in the grid are allowing some slightly hotter pixels to appear and that the differences in the center of the image are only caused by the colormap differences. A jet colormap is also really not the best way to compare these for this exact reason.

[djhoese]

Ok so I took the old file, set all values from i04 that were less than 338.6 to 338.5, then displayed it with matplotlib (what panoply uses I think) with the default viridis colormap and the color limits of 338.5 and 346.4. I couldn't even see the pixels that were in this range so I ran some python and found out these higher temperature pixels are only 4 in total. When I zoomed in to the spot on the matplotlib figure here's what we see right on the edge of the image (notice the x and y ticks, we're on the very left of the image):

These 4 high pixels are in the following (row, col) locations:

In [39]: np.argwhere(old_i04_data > 338.6)
Out[39]:
array([[796,   0],
       [797,   0],
       [797,   1],
       [798,   1]])

[djhoese]

This makes me think the grid is just slightly different or the EWA resampling bug fixes and changes are treating this pixel differently. Investigating...

[kathys]

Yeah, the new images are really similar. I have seen this on other HDF5 images that I made too. Not many pixels are different, but when they are, they are on the hot end of the BT scale.

[djhoese]

Are they always on the edge of the swath or edge of the image?

[djhoese]

Posting these images from our slack conversation for the record. The old data:

New data:

My hope this is caused by pytroll/pyresample#398 in the new code. The current line of testing is:

1. Check other data cases and if the differences are not on the edge of the data then my guess is wrong.
2. Take this existing data case, remap to a grid that includes more of the data to the left. If the result is a couple bright pixels then the new data is correct. If the result is bright group of pixels extending up and to the left then the old data is correct.

[kathys]

Okay Dave. I think you have convinced me. See new images with expanded grid:

[kathys]

bumi:data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/new_p2g_hdf2/noaa20_viirs_20210517_192444_v3_large.h5

bumi:/data/test_data/old_polar2grid_data/viirs_sdr_2021/cutouts/orig_p2g_hdf2/noaa20_viirs_20210517_192444_large.h5

[djhoese]

Ok I looked at these files and the only major differences are on the edges as expected based on the previously mentioned EWA PR and bug fix. There are small differences at random pixels of a min/max of 0.06 kelvin. I'm not sure what level of difference we're comfortable with, but for the randomly spread out pixels with that difference...I'm not concerned.

[djhoese]

As discussed on slack, the differences in these products are actually improvements in the EWA resampling that have been fixed in pyresample.