EnsembleStat : trouble reading ensemble files

[haksulee]

Could you help fix EnsembleStat.conf copied below?
the error message is :
ERROR : process_command_line() -> trouble reading ensemble file "/test/input/aggregated_data/aggregated_precip_v12_5day/20180118/precip_20180118_member00_day1_5.nc"

I tried combinations of different "name" and "level", e.g., Total Precipitation, tp, APCP, (,), (0,,) which all didn't work.
Is there a way to print out more debugging messages that suggest options I may try to fix the error ?
Related files are available at the google folder: https://drive.google.com/drive/folders/1DLSSwV3JEq-uqCigH7hGt_fz35lOpUOG

Config below is to read GEFSv12 5-day accumulated precipitation and AORC 5-day accumulated precipitation using EnsembleStat.

Thanks for the help.
Haksu

[config]

PROCESS_LIST = EnsembleStat

LOOP_BY = INIT
INIT_TIME_FMT = %Y%m%d%H
#INIT_BEG=2009123112
#INIT_END=2009123112
INIT_BEG=2018010400
INIT_END=2018010400
#INIT_END=2018022200

#INIT_INCREMENT=3600
INIT_INCREMENT=604800

#LEAD_SEQ = 24H
LEAD_SEQ = 120H
#LEAD_SEQ = begin_end_incr(120,120,0)

File I/O

#z FCST_ENSEMBLE_STAT_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst
#FCST_ENSEMBLE_STAT_INPUT_DIR = {INPUT_BASE}/sample_fcst
FCST_ENSEMBLE_STAT_INPUT_DIR = {INPUT_BASE}/aggregated_precip_v12_5day

#FCST_ENSEMBLE_STAT_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/arw-???-gep?/d01_{init?fmt=%Y%m%d%H}_0{lead?fmt=%HH}00.grib

#z FCST_ENSEMBLE_STAT_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/p25/ge???.t00z.pgrb2.0p25.f{lead?fmt=%3H}
FCST_ENSEMBLE_STAT_INPUT_TEMPLATE = {init?fmt=%Y%m%d}/precip_{init?fmt=%Y%m%d}_member??_day1_5.nc

#ENSEMBLE_STAT_CTRL_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_fcst
#ENSEMBLE_STAT_CTRL_INPUT_TEMPLATE = {init?fmt=%Y%m%d%H}/arw-fer-gep1/d01_{init?fmt=%Y%m%d%H}_0{lead?fmt=%HH}00.grib

#z OBS_ENSEMBLE_STAT_POINT_INPUT_DIR = {INPUT_BASE}/ascii2nc
#z OBS_ENSEMBLE_STAT_POINT_INPUT_TEMPLATE = precip24_{valid?fmt=%Y%m%d%H}.nc

#z OBS_ENSEMBLE_STAT_GRID_INPUT_DIR = {INPUT_BASE}/met_test/data/sample_obs/ST4
#z OBS_ENSEMBLE_STAT_GRID_INPUT_DIR = {INPUT_BASE}/sample_obs/ST4
#z OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE = ST4.{valid?fmt=%Y%m%d%H}.24h

#OBS_ENSEMBLE_STAT_GRID_INPUT_DIR = {INPUT_BASE}/aorc_regridded/{valid?fmt=%Y%m}
#OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE = AORC-OWP_{valid?fmt=%Y%m%d%H}z.nc4

OBS_ENSEMBLE_STAT_GRID_INPUT_DIR = {INPUT_BASE}/aggregated_precip_AORC_5day/{valid?fmt=%Y%m}
OBS_ENSEMBLE_STAT_GRID_INPUT_TEMPLATE = precip_{valid?fmt=%Y%m}_5day.nc

#ENSEMBLE_STAT_ENS_MEAN_INPUT_DIR =
#ENSEMBLE_STAT_ENS_MEAN_INPUT_TEMPLATE =

ENSEMBLE_STAT_CLIMO_MEAN_INPUT_DIR =
ENSEMBLE_STAT_CLIMO_MEAN_INPUT_TEMPLATE =

ENSEMBLE_STAT_CLIMO_STDEV_INPUT_DIR =
ENSEMBLE_STAT_CLIMO_STDEV_INPUT_TEMPLATE =

ENSEMBLE_STAT_OUTPUT_DIR = {OUTPUT_BASE}/ensemble
ENSEMBLE_STAT_OUTPUT_TEMPLATE = {init?fmt=%Y%m%d%H%M}/ensemble_stat

Field Info

#z MODEL = WRF
MODEL = GEFS
OBTYPE = MC_PCP

#FCST_VAR1_NAME = APCP
#FCST_VAR1_NAME = Total Precipitation
FCST_VAR1_NAME = tp

#z FCST_VAR1_LEVELS = A24
#zFCST_VAR1_LEVELS = Z0
#FCST_VAR1_LEVELS = (0,,)
FCST_VAR1_LEVELS = (,)

FCST_VAR1_OPTIONS = ens_ssvar_bin_size = 0.1; ens_phist_bin_size = 0.05;

#z OBS_VAR1_NAME = {FCST_VAR1_NAME}
#z OBS_VAR1_LEVELS = {FCST_VAR1_LEVELS}

OBS_VAR1_NAME = APCP_surface
#z OBS_VAR1_LEVELS = (0,,)
OBS_VAR1_LEVELS = (,)

OBS_VAR1_OPTIONS = {FCST_VAR1_OPTIONS}

EnsembleStat Settings

#LOG_ENSEMBLE_STAT_VERBOSITY = 2

#ENSEMBLE_STAT_N_MEMBERS = 6
ENSEMBLE_STAT_N_MEMBERS = 5

ENSEMBLE_STAT_CONFIG_FILE = {PARM_BASE}/met_config/EnsembleStatConfig_wrapped

ENSEMBLE_STAT_DESC = NA

ENSEMBLE_STAT_OBS_WINDOW_BEGIN = -5400
ENSEMBLE_STAT_OBS_WINDOW_END = 5400

ENSEMBLE_STAT_ENS_THRESH = 1.0

ENSEMBLE_STAT_VLD_THRESH = 1.0

ENSEMBLE_STAT_OUTPUT_PREFIX =

#ENSEMBLE_STAT_MET_OBS_ERR_TABLE =

ENSEMBLE_STAT_REGRID_TO_GRID = NONE
ENSEMBLE_STAT_REGRID_METHOD = NEAREST
ENSEMBLE_STAT_REGRID_WIDTH = 1
ENSEMBLE_STAT_REGRID_VLD_THRESH = 0.5
ENSEMBLE_STAT_REGRID_SHAPE = SQUARE
#ENSEMBLE_STAT_REGRID_CONVERT =
#ENSEMBLE_STAT_REGRID_CENSOR_THRESH =
#ENSEMBLE_STAT_REGRID_CENSOR_VAL =

ENSEMBLE_STAT_CENSOR_THRESH =
ENSEMBLE_STAT_CENSOR_VAL =

#ENSEMBLE_STAT_PROB_CAT_THRESH =
#ENSEMBLE_STAT_PROB_PCT_THRESH = ==0.25
#ENSEMBLE_STAT_ECLV_POINTS = 0.05

ENSEMBLE_STAT_MESSAGE_TYPE = ADPSFC
#ENSEMBLE_STAT_OBS_THRESH =
ENSEMBLE_STAT_DUPLICATE_FLAG = NONE
ENSEMBLE_STAT_SKIP_CONST = False

ENSEMBLE_STAT_OBS_ERROR_FLAG = FALSE

ENSEMBLE_STAT_ENS_SSVAR_BIN_SIZE = 1.0
ENSEMBLE_STAT_ENS_PHIST_BIN_SIZE = 0.05

#ENSEMBLE_STAT_CLIMO_MEAN_FILE_NAME =
#ENSEMBLE_STAT_CLIMO_MEAN_FIELD =
#ENSEMBLE_STAT_CLIMO_MEAN_REGRID_METHOD =
#ENSEMBLE_STAT_CLIMO_MEAN_REGRID_WIDTH =
#ENSEMBLE_STAT_CLIMO_MEAN_REGRID_VLD_THRESH =
#ENSEMBLE_STAT_CLIMO_MEAN_REGRID_SHAPE =
#ENSEMBLE_STAT_CLIMO_MEAN_TIME_INTERP_METHOD =
#ENSEMBLE_STAT_CLIMO_MEAN_MATCH_MONTH =
#ENSEMBLE_STAT_CLIMO_MEAN_DAY_INTERVAL = 31
#ENSEMBLE_STAT_CLIMO_MEAN_HOUR_INTERVAL = 6

#ENSEMBLE_STAT_CLIMO_STDEV_FILE_NAME =
#ENSEMBLE_STAT_CLIMO_STDEV_FIELD =
#ENSEMBLE_STAT_CLIMO_STDEV_REGRID_METHOD =
#ENSEMBLE_STAT_CLIMO_STDEV_REGRID_WIDTH =
#ENSEMBLE_STAT_CLIMO_STDEV_REGRID_VLD_THRESH =
#ENSEMBLE_STAT_CLIMO_STDEV_REGRID_SHAPE =
#ENSEMBLE_STAT_CLIMO_STDEV_TIME_INTERP_METHOD =
#ENSEMBLE_STAT_CLIMO_STDEV_MATCH_MONTH =
#ENSEMBLE_STAT_CLIMO_STDEV_DAY_INTERVAL = 31
#ENSEMBLE_STAT_CLIMO_STDEV_HOUR_INTERVAL = 6

ENSEMBLE_STAT_CLIMO_CDF_BINS = 1
ENSEMBLE_STAT_CLIMO_CDF_CENTER_BINS = False
ENSEMBLE_STAT_CLIMO_CDF_WRITE_BINS = True
#ENSEMBLE_STAT_CLIMO_CDF_DIRECT_PROB =

ENSEMBLE_STAT_MASK_GRID = FULL
ENSEMBLE_STAT_MASK_POLY =
MET_BASE/poly/HMT_masks/huc4_1605_poly.nc,
MET_BASE/poly/HMT_masks/huc4_1803_poly.nc,
MET_BASE/poly/HMT_masks/huc4_1804_poly.nc,
MET_BASE/poly/HMT_masks/huc4_1805_poly.nc,
MET_BASE/poly/HMT_masks/huc4_1806_poly.nc

ENSEMBLE_STAT_CI_ALPHA = 0.05

ENSEMBLE_STAT_INTERP_FIELD = BOTH
ENSEMBLE_STAT_INTERP_VLD_THRESH = 1.0
ENSEMBLE_STAT_INTERP_SHAPE = SQUARE
ENSEMBLE_STAT_INTERP_METHOD = NEAREST
ENSEMBLE_STAT_INTERP_WIDTH = 1

ENSEMBLE_STAT_OUTPUT_FLAG_ECNT = BOTH
ENSEMBLE_STAT_OUTPUT_FLAG_RPS = NONE
ENSEMBLE_STAT_OUTPUT_FLAG_RHIST = BOTH
ENSEMBLE_STAT_OUTPUT_FLAG_PHIST = BOTH
ENSEMBLE_STAT_OUTPUT_FLAG_ORANK = BOTH
ENSEMBLE_STAT_OUTPUT_FLAG_SSVAR = BOTH
ENSEMBLE_STAT_OUTPUT_FLAG_RELP = BOTH
#ENSEMBLE_STAT_OUTPUT_FLAG_PCT = BOTH
#ENSEMBLE_STAT_OUTPUT_FLAG_PSTD = BOTH
#ENSEMBLE_STAT_OUTPUT_FLAG_PJC = BOTH
#ENSEMBLE_STAT_OUTPUT_FLAG_PRC = BOTH
#ENSEMBLE_STAT_OUTPUT_FLAG_ECLV = BOTH

ENSEMBLE_STAT_NC_ORANK_FLAG_LATLON = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_MEAN = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_RAW = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_RANK = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_PIT = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_VLD_COUNT = TRUE
ENSEMBLE_STAT_NC_ORANK_FLAG_WEIGHT = FALSE

#ENSEMBLE_STAT_OBS_QUALITY_INC =
#ENSEMBLE_STAT_OBS_QUALITY_EXC =

#ENSEMBLE_STAT_ENS_MEMBER_IDS =
#ENSEMBLE_STAT_CONTROL_ID =

#ENSEMBLE_STAT_GRID_WEIGHT_FLAG =

[CPKalb]

Hi Haksu:

To get more debugging messages from MET, you would want to increase the LOG_MET_VERBOSITY. This verbosity defaults to a value of 2, but can be increased up to 5 for more information. I often start with 4 if I need to more logging information, as using 5 can be overwhelming sometimes. If the 4 doesn't provide enough information then I'll increase to 5.

LOG_MET_VERBOSITY = 4

For the netCDF file, I took a look at the precip_20180104_member00_day1_5.nc file from the Google Drive. Unfortunately, this file isn't technically in a format that MET can read. MET reads 3 types of netCDF files, CF-Compliant netCDF, netCDF output from wrf_pinterp, and files in MET's own format of netCDF. The requirements for reading netCDF can be found in section 4.1 of the MET users guide.

Although your file is not in a format that MET can read, it is very closely formatted to a CF compliant netCDF file. You can tell MET to try to read the file as a CF-Compliant netCDF and that may work. I typically test reading data using Plot-Data-Plane from the command line. This plots an image of how MET read the data so that we can verify that everything looks okay.

I first tried the following:

plot_data_plane precip_20180104_member00_day1_5.nc testmem.ps name="tp"; level="(0,*,*)";'

This doesn't work and I get the output message:

ERROR  : 
ERROR  : plot_data_plane -> file "precip_20180104_member00_day1_5.nc" not a valid data file
ERROR  : 

I then tried forcing MET to read this as a CF-Compliant File using the command:

plot_data_plane precip_20180104_member00_day1_5.nc testmem.ps name="tp"; level="(0,*,*)"; file_type=NETCDF_NCCF; '

This runs through MET and produces the following image.

Assuming the above image looks correct, you will need to add the file_type option to your Ensemble-Stat configuration to force MET to read this as a CF-Compliant file. Currently, we don't have support for adding the file type to Ensemble-Stat directly. However, you can use an override to get this to work. So, the following settings should be what you need to read the file:

FCST_VAR1_NAME = tp
FCST_VAR1_LEVELS = "(0,*,*)"
ENSEMBLE_STAT_MET_CONFIG_OVERRIDES = fcst = {file_type = NETCDF_NCCF;}

Those will enable you to read the first time in your file. However, I notice that there are two times in the files you sent over, 4-Jan 2018 and 9-Jan 2018. If you need to be able to distinguish between these times, you can set the levels as below:

FCST_VAR1_LEVELS = "({valid?fmt=%Y%m%d_%H%M%S},*,*)"

Does this work on your end?

Christina

[haksulee]

Thanks Christina

I used 3 lines:
FCST_VAR1_NAME = tp
FCST_VAR1_LEVELS = "(0,,)"
ENSEMBLE_STAT_MET_CONFIG_OVERRIDES = fcst = {file_type = NETCDF_NCCF;}

and I don't see the error.

I have a question.
For AORC data, should I also add something similar to ENSEMBLE_STAT_MET_CONFIG_OVERRIDES = fcst = {file_type = NETCDF_NCCF;} ?