Unable to do calibrate the performance sigma in ClimWIP recipe.

[DoMatte]

Hi,

I am struggling for a while now to calibrate the sigma using the ClimWIP recipe for my set of preproc.
i.e. commenting

performance_sigma: 0.43

and uncomment

calibrate_performance_sigma:
target_ref: tas_CLIM_reference
target: tas_CLIM_future

Note that the script works when I run it with a fixed performance sigma (0.43 for example, so not calibrating the sigma).
The log.txt shows that some of the weights are not computed, I thought it could be from a bad formulation from my recipe..but couldn'T find any mistake. It seem that the calibrate_performance_sigma is missing some data..

Any help would be greatly appreciated.

p.s the main_log_debug.txt, was too big but you have in attached the main_log.txt and the climwip/log.txt
pps. When I run the example recipe_climwip_brunner20esd.yml with the set-up for callibrating the performance sigma, it works. So, It might something in my script...

ClimWIP_Error.zip

[rswamina]

@Peter9192 , @stefsmeets , @lukasbrunner or @ruthlorenz - Can one of you help with this question?

[ruthlorenz]

The calibration fails in some circumstances, did you manage to produce a graph like this calibration graph ?
From the docu "Warning: It is highly recommended to visually inspect the graphical output of the calibration to check if everything worked as intended. In case the calibration fails, the best performance sigma will still be indicated in the figure (see example Fig. 77 below) but not automatically picked - the user can decide to use it anyway by setting it in the recipe (not recommenced)."

[DoMatte]

HI @ruthlorenz ,

No, that's the problem. The code run if I have a predetermine sigma, but failed to produce the calibration graph. Note the the prerpoc are done correctly. The error seems to come from calibrated_sigma.py/core_functions.py where this command failed
weights /= weights.sum(where=not_nan)
...
Any help would be very appreciated
Thanks in davance

[lukasbrunner]

Hi @DoMatte !

I'm at EGU this week so just briefly:

The problems seems to happen when calculating the baseline percentiles (so without weights):

ESMValTool/esmvaltool/diag_scripts/weighting/climwip/calibrate_sigmas.py

Lines 158 to 159 in 4d2e974

	percentiles_spread, inside_ratio = calculate_percentiles(
	target, weights_matrix, percentiles, weighted=False)

As I understand the message, the problem is, that the array is empty at this point:

ESMValTool/esmvaltool/diag_scripts/weighting/climwip/core_functions.py

Line 297 in 4d2e974

min_val = weighted_quantiles.min()

Best bet is because it was full of nans here:

ESMValTool/esmvaltool/diag_scripts/weighting/climwip/core_functions.py

Lines 282 to 285 in 4d2e974

	# remove nans
	not_nan = np.where((np.isfinite(values) & np.isfinite(weights)))[0]
	values = values[not_nan]
	weights = weights[not_nan]

It is strange how this can happen. If I remember correctly there should be map output for all diagnostics from the tool? Can you have a look at the maps for the target, do they look fine?

One thing that I noticed is that you are using a custom shapefile, I've never done that myself, so maybe that is causing the problem? In particular it could be the area weighted mean calculation (that is only needed for sigma calibration not for anything else I think, so it would only appear here):

ESMValTool/esmvaltool/diag_scripts/weighting/climwip/core_functions.py

Line 21 in 4d2e974

means = data_array.weighted(weights_lat).mean(dim=['lat', 'lon'])

Maybe the shape of the polygon is somehow causing this? Note that "weights must be a DataArray and cannot contain missing values. Missing values can be replaced by weights.fillna(0)."

https://docs.xarray.dev/en/stable/generated/xarray.DataArray.weighted.html

Can you try checking these two things? Otherwise I can look in more detail next week.

Cheers,
Lukas

[DoMatte]

Hi @lukasbrunner ,
thanks for your help. I am also out for the week. I check this out next week and come back to you.
Thanks again and enjoy EGU