recipe_flato13ipcc.yml (figs. 9.6 and 9.14) broken #1724
Comments
|
Fig. 9.6 is pattern correlation, not perfmetrics. I think this was implemented by @LisaBock |
This should be fixed in the |
|
The 9.6. thing seems to be a preprocessor problem though with "failed to concatenate cubes". Does the log name a specific model or is it the multi-model mean that recently had some changes with its time handling? |
|
@axel-lauer could it be that you tested this with ESMValCore |
|
I tested with v2.0.0 (just double-checked). I am now retesting figs. 9.6 and 9.14 with the latest fixes... |
|
Pulling the lastest fixes of ESMValCore v2.0.0 solves the problem with fig. 9.14. The error "different calendars" in fig. 9.6 persists, however, this time for variable rlut. |
Doesn't seem to be a specific model, here is the log file. |
|
Just tested 9.6. myself, ran fine. Did you remember that CMIP3 cannot be run with parallel processing? ESMValGroup/ESMValCore#430 |
|
Thanks @bettina-gier for recalling this! Maybe we should add a comment in the recipe about this. |
I used only 1 task as the recipe gets "stuck" before doing any preprocessing when trying to use 2 or more tasks. I'd love to reproduce your success, but doesn't work for me. |
|
Hmm interesting. Try running it with CMIP5 only just as a test? |
|
@bettina-gier: Successfully tested figure 9.6 with CMIP5 data only. This suggests the problem is caused by CMIP3 models, which is strange as I am using the data located in our standard directory "ESMValTool2/CMIP3" on Mistral. |
|
I tested it first with "/work/kd0956/CMIP3/" on Mistral, reran it with the directory you used and it worked fine. Using the latest master branch in the core. Retested again with the v2.0.0 core branch and that ran through as well. |
|
I tested it now also with ESMValTool |
|
I have this path to CMIP3 in our local data pool: |
|
@mattiarighi I used /work/bd0854/DATA/ESMValTool2/CMIP3 which is a symbolic link pointing to the path you mentioned. Since the diagnostic works for @bettina-gier and @LisaBock , I would propose to mark flato13ipcc.yml as tested successfully in #1697. |
|
But then we should remove the CMIP3 data in our datapool (ESMValTool1 and, as link, ESMValTool2) and replace them with links to If I understand correctly, the latter is the path successfully used by @bettina-gier and @LisaBock for testing Fig. 9.6, right? |
|
I tested it again with |
|
Ok, let me try 9.6 and the we can close this hopefully. |
|
My runs took 25-30 min on mistral each, with both cores and both paths. |
|
After one of the latest updates it was also necessary to run |
|
Tested successfully. |
Testing of the following figures in recipe_flato13ipcc.yml (ESMValTool master branch) with ESMValCore v2.0.0 (https://github.com/ESMValGroup/ESMValCore/tree/v2.0.0) failed:
All other diagnostics in recipe_flato13ipcc.yml have been tested successfully and look fine.
Errors:
Error message fig. 9.6:
ERROR [7867] Failed to run concatenate([<iris 'Cube' of air_temperature / (K) (time: 1800; latitude: 60; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 1488; latitude: 46; longitude: 72)>, <iris 'Cube' of air_temperature / (K) (time: 1680; latitude: 90; longitude: 144)>, <iris 'Cube' of air_temperature / (K) (time: 1812; latitude: 60; longitude: 90)>, <iris 'Cube' of air_temperature / (K) (time: 1440; latitude: 46; longitude: 72)>, <iris 'Cube' of air_temperature / (K) (time: 1680; latitude: 90; longitude: 144)>, <iris 'Cube' of air_temperature / (K) (time: 1212; latitude: 160; longitude: 320)>, <iris 'Cube' of air_temperature / (K) (time: 600; latitude: 64; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 1200; latitude: 64; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 1812; latitude: 64; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 1812; latitude: 48; longitude: 96)>, <iris 'Cube' of air_temperature / (K) (time: 120; latitude: 128; longitude: 256)>, <iris 'Cube' of air_temperature / (K) (time: 120; latitude: 128; longitude: 256)>, <iris 'Cube' of air_temperature / (K) (time: 1812; latitude: 64; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 1692; latitude: 72; longitude: 96)>, <iris 'Cube' of air_temperature / (K) (time: 1680; latitude: 145; longitude: 192)>, <iris 'Cube' of air_temperature / (K) (time: 1680; latitude: 73; longitude: 96)>, <iris 'Cube' of air_temperature / (K) (time: 1572; latitude: 160; longitude: 320)>, <iris 'Cube' of air_temperature / (K) (time: 1320; latitude: 64; longitude: 128)>, <iris 'Cube' of air_temperature / (K) (time: 2892; latitude: 96; longitude: 192)>, <iris 'Cube' of air_temperature / (K) (time: 1560; latitude: 96; longitude: 192)>, <iris 'Cube' of air_temperature / (K) (time: 1560; latitude: 45; longitude: 72)>, <iris 'Cube' of air_temperature / (K) (time: 1440; latitude: 46; longitude: 72)>, <iris 'Cube' of air_temperature / (K) (time: 1200; latitude: 64; longitude: 128)>], {}) INFO [7867] Maximum memory used (estimate): 0.3 GB INFO [7867] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ERROR [7867] Program terminated abnormally, see stack trace below for more information Traceback (most recent call last): File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 428, in run fire.Fire(ESMValTool()) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 138, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire target=component.__name__) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 405, in run process_recipe(recipe_file=recipe, config_user=cfg) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 98, in process_recipe recipe.run() File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_recipe.py", line 1359, in run max_parallel_tasks=self._cfg['max_parallel_tasks']) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 641, in run_tasks _run_tasks_sequential(tasks) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 653, in _run_tasks_sequential task.run() File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 238, in run input_files.extend(task.run()) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 238, in run input_files.extend(task.run()) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 242, in run self.output_files = self._run(input_files) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 428, in _run product.apply(step, self.debug) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 296, in apply self.cubes = preprocess(self.cubes, step, **self.settings[step]) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 238, in preprocess result.append(_run_preproc_function(function, items, settings)) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 224, in _run_preproc_function return function(items, **kwargs) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/_io.py", line 185, in concatenate cubes = sorted(cubes, key=lambda c: c.coord("time").cell(0).point) File "cftime/_cftime.pyx", line 1304, in cftime._cftime.datetime.__richcmp__ TypeError: cannot compare cftime.DatetimeNoLeap(1861-01-16 12:00:00) and real_datetime(1850, 1, 16, 12, 0) (different calendars)Error message fig. 9.14:
ERROR [32159] Failed to run climate_statistics(sea_surface_temperature (time: 252; cell index along second dimension: 300; cell index along first dimension: 360) Dimension coordinates: time x - - cell index along second dimension - x - cell index along first dimension - - x Auxiliary coordinates: latitude - x x longitude - x x Attributes: Conventions: CF-1.4 associated_files: baseURL: http://cmip-pcmdi.llnl.gov/CMIP5/dataLocation gridspecFile: gridspec_ocean_fx_ACCESS1-0_historical_r0i0p0.nc... branch_time: 109207.0 cmor_version: 2.8.0 comment: "this may differ from ""surface temperature"" in regions of sea ice." contact: The ACCESS wiki: http://wiki.csiro.au/confluence/display/ACCESS/Home. Contact... experiment: historical experiment_id: historical forcing: GHG, Oz, SA, Sl, Vl, BC, OC, (GHG = CO2, N2O, CH4, CFC11, CFC12, CFC113,... frequency: mon initialization_method: 1 institute_id: CSIRO-BOM institution: CSIRO (Commonwealth Scientific and Industrial Research Organisation, Australia),... model_id: ACCESS1-0 modeling_realm: ocean parent_experiment: pre-industrial control parent_experiment_id: piControl parent_experiment_rip: r1i1p1 physics_version: 1 product: output project_id: CMIP5 realization: 1 references: See http://wiki.csiro.au/confluence/display/ACCESS/ACCESS+Publications source: ACCESS1-0 2011. Atmosphere: AGCM v1.0 (N96 grid-point, 1.875 degrees EW... table_id: Table Omon (27 April 2011) 694b38a3f68f18e58ba80230aa4746ea title: ACCESS1-0 model output prepared for CMIP5 historical version_number: v20120115 Cell methods: mean: time, {}) INFO [32159] Maximum memory used (estimate): 0.3 GB INFO [32159] Sampled every second. It may be inaccurate if short but high spikes in memory consumption occur. ERROR [32159] Program terminated abnormally, see stack trace below for more information Traceback (most recent call last): File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 428, in run fire.Fire(ESMValTool()) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 138, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 468, in _Fire target=component.__name__) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/fire/core.py", line 672, in _CallAndUpdateTrace component = fn(*varargs, **kwargs) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 405, in run process_recipe(recipe_file=recipe, config_user=cfg) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_main.py", line 98, in process_recipe recipe.run() File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_recipe.py", line 1359, in run max_parallel_tasks=self._cfg['max_parallel_tasks']) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 641, in run_tasks _run_tasks_sequential(tasks) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 653, in _run_tasks_sequential task.run() File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 238, in run input_files.extend(task.run()) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/_task.py", line 242, in run self.output_files = self._run(input_files) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 428, in _run product.apply(step, self.debug) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 296, in apply self.cubes = preprocess(self.cubes, step, **self.settings[step]) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 241, in preprocess result.append(_run_preproc_function(function, item, settings)) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/__init__.py", line 224, in _run_preproc_function return function(items, **kwargs) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/_time.py", line 399, in climate_statistics time_weights = get_time_weights(cube) File "/mnt/lustre02/work/bd0854/b380103/ESMValCore/esmvalcore/preprocessor/_time.py", line 176, in get_time_weights time_weights = time_thickness * da.ones_like(cube.data) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/dask/array/core.py", line 1221, in __array_ufunc__ return elemwise(numpy_ufunc, *inputs, **kwargs) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/dask/array/core.py", line 3912, in elemwise broadcast_shapes(*shapes) File "/mnt/lustre02/work/bd0854/b380103/miniconda3/envs/esmvaltool/lib/python3.7/site-packages/dask/array/core.py", line 3873, in broadcast_shapes "shapes {0}".format(" ".join(map(str, shapes))) ValueError: operands could not be broadcast together with shapes (252,) (252, 300, 360)The text was updated successfully, but these errors were encountered: