Enhance regrid_data_plane to process Smoke and Dust from GOES-16 ADP datasets.

[JohnHalleyGotway]

This is a request from the NOAA/EMC Aerosols group. The regrid_data_plane tool was enhanced to process GOES-16 AOD data. During a meeting at NCWCP on 9/11/2019, Ho-Chun Huang requested that regrid_data_plane be enhanced to also process data from GOES-16 ADP files.

See a description of the GOES-16 ADP data format attached.
GOES-16_ADP_readme_idl_code_20180328.docx

Also see and excerpt from the ADP user's guide attached.
GOES-16 ADP Users Guide extract.pdf

The ADP files are very similar to the AOD files, but slightly different. The relevant data variables are named Aerosol, Dust, and Smoke. Each of these fields contains 0's or 1's as a mask to define whether Aerosol, Dust, or Smoke is present at that pixel. In addition, the DQF variable contains quality control flags. However, the QC info is stored in the 8 bits of the DQF values as described in the document above.

When processing Smoke or Dust, parse the DCF bits to construct a QC value of 0, 1, or 2, for high, medium, and low quality, respectively. Use the user-specified -qc flag command line option to determine which pixels should be used. When doing the regridding step, determine how many of the pixels inside each grid box meet the -qc flag setting. And then report the output value for that grid box as the average value of the good data pixel values. The result should be a number between 0 and 1 indicating the percentage of the pixels in that grid box containing smoke or dust.

Ultimately, Ho-Chun plans to process both the ADP files to compute fields for smoke percentage and dust percentage. Then he'll use those percentages to define masks when computing stats in Grid-Stat for the GOES-16 AOD values. For example, he can compute the RMSE for AOD for grid boxes that are at least 50% smoke.

Even though Ho-Chun has no planned usage for the Aerosol field, we may as well include that as an optional output.

Here's a description of how the -valid_thresh command line option of regrid_data_plane should be used. Let's say one target output grid box has 100 GOES-16 pixels inside of it. The -valid_thresh option defines the ratio (between 0 and 1) of those 100 pixels which must meet the requested -qc flag information in order to get an output value for that grid box.

Sample data can be found on WCOSS, here:
The daily ADP data file can be found in (phase II)
/gpfs/gd3/emc/meso/noscrub/Ho-Chun.Huang/GOES16_AOD/ADP/yyyymmdd

module load NetCDF/4.2/serial

You can use ncdump to read the header, I believe we will read Smoke, Dust, and DQF (for high, medium, and low confidence of Smoke and Dust mask)

ncdump -h OR_ABI-L2-ADPC-M6_G16_s20192472341118_e20192472343491_c20192472344453.nc | more

[Ho-ChunHuang-NOAA]

I discussed the GOES-16 ADP product content with NESDIS/aerosol validation group (Pubu Ciren) on September 18. The meaning of the Aerosol/Smoke/Dusk mask are different from my previous understanding.

The Aeorosl Mask is NOT an indicator of the AOD retrieval of "ALL" aerosol nor "Aerosol other than Smoke and Dust".

The Aerosol mask = 1 if [Smoke=1 OR Dust=1]. Thus there is no information of any pixel that has aerosol other than smoke and dust (such as anthropocentric aerosol) provided by ADP products. It is impossible to compute the fraction of Smoke AOD or Dust AOD over total AOD for each mapping grid.

NESDIS GOES-16 ADP product can only identify which ABI (sensor) pixel likely are all smoke/dust particle in its footprint, with confident level determined by the quality flag [qf] (bit 2-3 for smoke and bit 4-5 for dust). Since Smoke/Dust AOD is the final OBS (what we want) to be compared to same modeled variable, thus it may need to be read in by regrid_data_plane as additional constraint to find the AOD pixel to be gridded to selected model grid.

Note we already have total AOD qf to control the AOD mapping in input GOES-16 AOD file. This function should be retained because of the need to other broad community users. For NCEP operational need as a special case, to find the mapped Smoke AOD in model grid, we need to add additional condition of

High quality smoke AOD (-qc 0) = AOD with [high AOD qf]+[high Smoke confidence qf], i.e., total AOD qf <= 0 && smoke mask qf <=0.

Medium quality smoke AOD (-qc 1) = AOD with [high AOD qf]+[medium Smoke confidence qf] && [medium AOD qf]+[high Smoke confidence qf] && [medium AOD qf]+[medium Smoke confidence qf]. i.e., total AOD qf <= 1 && smoke mask qf <= 1 ??

And

Low quality Smoke AOD is total AOD qf <= 2 && smoke mask qf <= 2, ??

I do not know whether each pair occurred or not in reality, I need you expertise to refine it if you like to pursuit this approach.

I still want to use current regrid_data_plane function for total AOD comparison to that of NAQFC/CMAQ or FV3GFS-CHEM (or GEFS-aerosol).

For the Smoke AOD mapping I would like to create to be compared to HYSPLIT/HRRR-smoke smoke AOD output

Using -qc 0, -qc 0,1 and -qc 0,1,2.

I attached a new ADP description from NESDIS USer's Guide here

GOES-16 ADP Users Guide extract.pdf

Full version of the users guide can be found in IDL readme docs in John's comment.

[Ho-ChunHuang-NOAA]

The users of Smoke/Dust AOD (for now) probably are NCEP and ESRL (HRRR-smoke group). The other satellite are VIIRS AOD and ADP.

METplus's algorithm need to be as general as possible to operate on other satellite inputs. We can keep the current option input for regular regrid_data_plane users, but use several environmental settings to tell the algorithm whether it needs additional constraint, where is the input file (i.e., ADP file) , and what is the parameters to be use as a criteria "Smoke mask" or "Dust mask".

[JohnHalleyGotway]

When you run regrid_data_plane with AOD input, you can request:

• All AOD in the output (name="AOD")
• AOD Smoke in the output... this requires ADP to be processed as well (name="AOD_Smoke")
• AOD Dust in the output... this requires ADP to be processed as well (name="AOD_Dust")

Need a way to specify the ADP file to be processed, either a command line option or environment variable.

Here are two matching AOD/ADP file names:
/gpfs/gd3/emc/meso/noscrub/Ho-Chun.Huang/GOES16_AOD/ADP/20190919/OR_ABI-L2-ADPC-M6_G16_s20192622351163_e20192622353536_c20192622354379.nc
/gpfs/gd3/emc/meso/noscrub/Ho-Chun.Huang/GOES16_AOD/AOD/20190919/OR_ABI-L2-AODC-M6_G16_s20192622356163_e20192622358536_c20192630000280.nc

The start (s) and end (e) times must match but the creation time (c) will not.

If the s and e times differs between AOD and ADP, then error out.

Use the logic Ho-Chun described previously to derive AOD_Smoke or AOD_Dust.

If AOD_Smoke is requested, for each pixel...
(1) Get AOD value from AOD file.
(2) Get AOD QF and the smoke mask value (0 or 1).
(3) If smoke_mask = 0, reset AOD value to missing.
(4) If smoke_mask = 1, check the AOD QF and the Smoke mask QF from ADP file.
(5) If QF's are acceptable, include this Smoke AOD pixel.
(6) Apply the same logic for all pixels in the grid box and compute the mean (or other user choice) of all valid pixels.

Employ similar logic for Dust.

[Ho-ChunHuang-NOAA]

export MET_GEOSTATIONARY_DATA=/meso2/noscrub/Ho-Chun.Huang/GOES16_AOD/g16_conus_latlon_2km_20180620.dat

export MET_TMP_DIR=/stmpp2/Ho-Chun.Huang/METPLUS_TMP_${mdl_name}/${NOW}

if [ -d ${MET_TMP_DIR} ]; then
/bin/rm -f ${MET_TMP_DIR}/*
else
mkdir -p ${MET_TMP_DIR}
fi

case ${i} in
high) value="0"
;;
medium) value="0,1"
;;
low) value="0,1,2"
;;
med_only) value="1"
;;
low_only) value="2"
;;
*) echo "qc_flag=${i} is not defined"
exit;;
esac

• aod_file is the GOES-16 AOD file, e.g., files in /meso2/noscrub/Ho-Chun.Huang/GOES16_AOD/AOD/yyyymmdd
• mod_file is the model_aot_grib2 file, e.g., mod_file=/naqfc/save/Ho-Chun.Huang/plot/cmaq/parm/aqm.t12z.aot.f15.148.grib2
• output file is the model grid mapped file, e.g., nc file in /meso2/noscrub/${USER}/GOES16_AOD/REGRID/aqm.20190919/OBS_AOD_aqm_g16_20190919_12_high.nc

regrid_data_plane ${aod_file} ${mod_file} ${output_file} -field 'name="AOD"; level="(,)";' -method UW_MEAN -v 1 -qc ${value}

[hsoh-u]

Checked in to feature_1194_adp

[hsoh-u]

Merged to develop branch

[Ho-ChunHuang-NOAA]

The description of the smoke and dust mask confidence level has been updated

(OLD)
; bit 6 (sunglint)
tmp_mask=DQF
idx = WHERE(((tmp_mask AND 64) EQ 64), COMPLEMENT=cidx, nc)
IF nc GT 0 THEN smoke_con[idx] = 0
IF nc GT 0 THEN dust_con[idx] = 0

;Byte1, bit7 (solar and satellite angles)
tmp_mask=DQF
idx = WHERE(((tmp_mask AND 128) EQ 128), COMPLEMENT=cidx, nc)
IF nc GT 0 THEN dust_con[idx] = 0

(NEW and Correct one)
; bit 6 (sunglint)
tmp_mask=DQF
idx = WHERE(((tmp_mask AND 64) EQ 64), COMPLEMENT=cidx, nc)
IF nc GT 0 THEN dust_con[idx] = 0

;Byte1, bit7 (solar and satellite angles)
tmp_mask=DQF
idx = WHERE(((tmp_mask AND 128) EQ 128), COMPLEMENT=cidx, nc)
IF nc GT 0 THEN dust_con[idx] = 0
IF nc GT 0 THEN smoke_con[idx] = 0

Please update MET9.0 beta version before official release.

I aslo want to point out that from AOD files
AOD high quality is quality flag=0
AOD medium quality is quality flag=0,1
AOD low quality is quality flag=0,1,2

For ADP files
Smoke Mask high confidence flag=3
Smoke Mask medium confidence flag=2
Smoke Mask low confidence flag=1

Dust Mask is the same as Smoke Mask