stat_analysis for generating aggregated RMSE

[jir245]

Hi,

I'm trying to use stat_analysis to generate RMSE statistics that are aggregated over a time range. I would like to run stat_analysis on output files from ensemble_stat as well as on output files from grid_stat in a separate evaluation. I'm having a difficult time figuring out what is the right way to produce RMSE.

For grid stat, I have tried the following, but wasn't getting what I had hoped:
stat_analysis -lookin ls /filepath/*cnt.txt -job aggregate_stat -line_type sl1l2 -out_line_type cnt -by VX_MASK
stat_analysis -lookin ls /filepath/*cnt.txt -job aggregate -line_type cnt -by VX_MASK

For ensemble stat output, I have had more success getting RMSE as an output, but I want to make sure this is the correct approach:
stat_analysis -lookin ls /filepath/*ecnt.txt -job aggregate -line_type ecnt -by VX_MASK

Any help on the correct approach for producing RMSE output would be very helpful,
Thank you

[CPKalb]

Hi:

The first command (stat_analysis -lookin ls /filepath/*cnt.txt -job aggregate_stat -line_type sl1l2 -out_line_type cnt -by VX_MASK), looks like it's only searching for cnt files. However, the command is performing an aggregate_stat from SL1L2 lines. In this case, there would be no input SL1L2 lines (since only the cnt lines are passed in). To get output, you could change the file search to say "ls /filepath/*sl1l2.txt"... or just switch to "-lookin /filepath". In this second case, stat_analysis will search the directory and process each .stat file it finds.

Thanks,
Christina

[jir245]

Hi Christina,

Sorry, I typed that in wrong. I did point to the sl1l2.txt files in the first command that I posted. It does produce output, only MAE. I'm trying to figure out how to get RMSE as an output.

Thanks for your help,
Juli

[CPKalb]

Hi Juli:

If it's producing MAE, then it should be outputting a CNT line. What does your out for RMSE look like? Is it completely missing or not as expected? Would you be able to post your output here or send it to us so I can take a closer look?

Thanks,
Christina

[jir245]

Hi Christina, I think I did get the RMSE output to work for stat_analysis using the sl1l2 grid_stat output files. Not sure what I did differently the first time. Can you let me know what the right way to aggregate and generate RMSE values from ensemble_stat output would be? Looking back at what I did, I was able to generate some statistics with stat_analysis aggregate on ecnt line type, including CRPS, but RMSE columns are empty. In the MET documentation, I only see an aggregate option on ECNT output, I don't see a aggregate_stat option. Any help would be greatly appreciated, Juli

…

On Thu, May 26, 2022 at 8:04 AM Christina Kalb ***@***.***> wrote: Hi Juli: If it's producing MAE, then that means it is outputting a CNT line. What does your out for RMSE look like? Is it completely missing or not as expected? It might help to see a line if you are able to post it here. Thanks, Christina — Reply to this email directly, view it on GitHub <#1632 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AYBBMFII26JVBUHOEZQQMB3VL6HHBANCNFSM5W5X36RQ> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[CPKalb]

Hi Juli:

Running aggregate using the ecnt line is the correct way to aggregate RMSE for ensembles. However, @JohnHalleyGotway has tested this, and found that both ME and RMSE are always coming out as 0 even when they should not be. So, this is likely a bug. We have an issue written up to fix this bug. Thanks for letting us know about this isse!

Christina

[jir245]

Hi Christina,

When running stat_analysis on grid_stat output (sl1l2 to generate cnt stats), how do I generate the bootstrapped confidence intervals for stats such as RMSE, MAE? It currently comes up as NA.

Thanks

[CPKalb]

Hi Juli:

Stat analysis doesn't bootstrap the aggregation of SL1L2 lines to CNT, which is why your output is NA. We have an issue out to address this. However, the issue has been around for a long time. If you have time to add a comment to the issue on how useful this would be, it may help bump the priority higher.

Right now, METviewer can perform this aggregation if you are able to load the data to a database.

Christina