Impossible to download Earth-scale data

[davide-f]

Description

While improving PyPSA-Africa towards PyPSA-Earth, I've noticed that when I try to create a cutout using atlite, atlite returns an error because copernicus is not able to convert grib files into netCDF format, as requested by the atlite package, and the workflow crashes.

Expected Behavior

Atlite shall be able to derive the desiderable cutout for the Earth.
The error message below can be reproduced by using the following code:

import atlite
cutout = atlite.Cutout(path="world_2013.nc",
                       module="era5",
                       x=slice(-179.4, 179.7),
                       y=slice(-59.1, 87.3),
                       time="2013"
                       )
cutout.prepare()

Actual Behavior

The workflow stops because Atlite is not able to successfully download the data from copernicus.
The raw grib files seem to be procuced but they cannot be successfully converted into netCDF format, hence the procedure stops.
When creating the cutout for the world exactly :"area": [87.3, -179.4, -59.1, 179.7] in copernicus.

Error Message

The request you have submitted is not valid

Reason:  grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints Cannot create netCDF classic format, dataset is too large! Try splitting the input GRIB(s). grib_to_netcdf: Version 2.24.2 grib_to_netcdf: Processing input file '/cache/tmp/8599e6f4-dc3e-423e-a66e-72e4d44ac365-adaptor.mars.internal-1645467199.6672807-10228-8-tmp.grib'. grib_to_netcdf: Found 17520 GRIB fields in 1 file. grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate grib_to_netcdf: Creating netCDF file '/cache/data3/adaptor.mars.internal-1645468582.836742-10228-15-8599e6f4-dc3e-423e-a66e-72e4d44ac365.nc' grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $ grib_to_netcdf: Creating large (64 bit) file format. grib_to_netcdf: Defining variable 't2m'. grib_to_netcdf: Defining variable 'stl4'.

Your Environment

• The atlite version used: 0.2.5
• How you installed atlite (conda, pip or github): conda
• Operating System: Linux
• My environment: CentOS Linux
  (output of `conda list`)

  ``` # Name Version Build Channel

_libgcc_mutex 0.1 conda_forge conda-forge
_openmp_mutex 4.5 1_gnu conda-forge
affine 2.3.0 py_0 conda-forge
alsa-lib 1.2.3 h516909a_0 conda-forge
ampl-mp 3.1.0 h2cc385e_1005 conda-forge
amply 0.1.4 py_0 conda-forge
anyio 3.4.0 py39hf3d152e_0 conda-forge
appdirs 1.4.4 pyh9f0ad1d_0 conda-forge
argon2-cffi 21.1.0 py39h3811e60_2 conda-forge
async_generator 1.10 py_0 conda-forge
atlite 0.2.5 pyhd8ed1ab_0 conda-forge
attrs 21.2.0 pyhd8ed1ab_0 conda-forge
babel 2.9.1 pyh44b312d_0 conda-forge
backcall 0.2.0 pyh9f0ad1d_0 conda-forge
backports 1.0 py_2 conda-forge
backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge
beautifulsoup4 4.10.0 pyha770c72_0 conda-forge
bleach 4.1.0 pyhd8ed1ab_0 conda-forge
blosc 1.21.0 h9c3ff4c_0 conda-forge
bokeh 2.4.2 py39hf3d152e_0 conda-forge
boost-cpp 1.74.0 h312852a_4 conda-forge
bottleneck 1.3.2 py39hce5d2b2_5 conda-forge
branca 0.4.2 pyhd8ed1ab_0 conda-forge
brotli 1.0.9 h7f98852_6 conda-forge
brotli-bin 1.0.9 h7f98852_6 conda-forge
brotlipy 0.7.0 py39h3811e60_1003 conda-forge
bzip2 1.0.8 h7f98852_4 conda-forge
c-ares 1.18.1 h7f98852_0 conda-forge
ca-certificates 2021.10.8 ha878542_0 conda-forge
cairo 1.16.0 h6cf1ce9_1008 conda-forge
cartopy 0.20.1 py39ha48bd0d_1 conda-forge
cdsapi 0.5.1 pyhd8ed1ab_0 conda-forge
certifi 2021.10.8 py39hf3d152e_1 conda-forge
cffi 1.15.0 py39h4bc2ebd_0 conda-forge
cfgv 3.3.1 pyhd8ed1ab_0 conda-forge
cfitsio 3.470 hb418390_7 conda-forge
cftime 1.5.1.1 py39hce5d2b2_1 conda-forge
charset-normalizer 2.0.8 pyhd8ed1ab_0 conda-forge
click 7.1.2 pyh9f0ad1d_0 conda-forge
click-plugins 1.1.1 py_0 conda-forge
cligj 0.7.2 pyhd8ed1ab_1 conda-forge
cloudpickle 2.0.0 pyhd8ed1ab_0 conda-forge
coincbc 2.10.5 hcee13e7_1 conda-forge
colorama 0.4.4 pyh9f0ad1d_0 conda-forge
colorcet 2.0.6 pyhd8ed1ab_0 conda-forge
conda 4.11.0 py39hf3d152e_0 conda-forge
conda-package-handling 1.7.3 py39h3811e60_1 conda-forge
configargparse 1.5.3 pyhd8ed1ab_0 conda-forge
connection_pool 0.0.3 pyhd3deb0d_0 conda-forge
contextily 1.2.0 pyhd8ed1ab_0 conda-forge
country_converter 0.7.4 pyhd8ed1ab_0 conda-forge
countrycode 0.2 pypi_0 pypi
cryptography 36.0.0 py39h95dcef6_0 conda-forge
curl 7.80.0 h2574ce0_0 conda-forge
cycler 0.11.0 pyhd8ed1ab_0 conda-forge
cytoolz 0.11.2 py39h3811e60_1 conda-forge
dask 2021.3.1 pyhd8ed1ab_0 conda-forge
dask-core 2021.3.1 pyhd8ed1ab_0 conda-forge
datashader 0.13.0 pyh6c4a22f_0 conda-forge
datashape 0.5.4 py_1 conda-forge
datrie 0.8.2 py39h3811e60_3 conda-forge
dbus 1.13.6 h48d8840_2 conda-forge
debugpy 1.5.1 py39he80948d_0 conda-forge
decorator 5.1.0 pyhd8ed1ab_0 conda-forge
defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge
deprecation 2.1.0 pyh9f0ad1d_0 conda-forge
descartes 1.1.0 py_4 conda-forge
distlib 0.3.3 pyhd8ed1ab_0 conda-forge
distributed 2021.4.1 py39hf3d152e_1 conda-forge
docutils 0.18.1 py39hf3d152e_0 conda-forge
editdistance-s 1.0.0 py39h1a9c180_2 conda-forge
entrypoints 0.3 pyhd8ed1ab_1003 conda-forge
entsoe-py 0.3.7 pyhd8ed1ab_0 conda-forge
esy-osm-pbf 0.0.1 pypi_0 pypi
esy-osmfilter 1.0.11 pypi_0 pypi
et_xmlfile 1.0.1 py_1001 conda-forge
expat 2.4.1 h9c3ff4c_0 conda-forge
filelock 3.4.0 pyhd8ed1ab_0 conda-forge
fiona 1.8.20 py39h427c1bf_2 conda-forge
folium 0.12.1.post1 pyhd8ed1ab_0 conda-forge
font-ttf-dejavu-sans-mono 2.37 hab24e00_0 conda-forge
font-ttf-inconsolata 3.000 h77eed37_0 conda-forge
font-ttf-source-code-pro 2.038 h77eed37_0 conda-forge
font-ttf-ubuntu 0.83 hab24e00_0 conda-forge
fontconfig 2.13.1 hba837de_1005 conda-forge
fonts-conda-ecosystem 1 0 conda-forge
fonts-conda-forge 1 0 conda-forge
fonttools 4.28.2 py39h3811e60_0 conda-forge
freetype 2.10.4 h0708190_1 conda-forge
freexl 1.0.6 h7f98852_0 conda-forge
fsspec 2021.11.0 pyhd8ed1ab_0 conda-forge
gdal 3.3.3 py39h0494519_0 conda-forge
gdrive 2.1.0 haa36a5b_0 conda-forge
geographiclib 1.52 pyhd8ed1ab_0 conda-forge
geopandas 0.10.2 pyhd8ed1ab_0 conda-forge
geopandas-base 0.10.2 pyha770c72_0 conda-forge
geoplot 0.4.4 pyhd8ed1ab_0 conda-forge
geopy 2.2.0 pyhd8ed1ab_0 conda-forge
geos 3.9.1 h9c3ff4c_2 conda-forge
geotiff 1.7.0 hcfb7246_3 conda-forge
geoviews 1.9.2 pyhd8ed1ab_0 conda-forge
geoviews-core 1.9.2 pyha770c72_0 conda-forge
gettext 0.19.8.1 h73d1719_1008 conda-forge
giflib 5.2.1 h36c2ea0_2 conda-forge
gitdb 4.0.9 pyhd8ed1ab_0 conda-forge
gitpython 3.1.24 pyhd8ed1ab_0 conda-forge
glib 2.70.1 h780b84a_0 conda-forge
glib-tools 2.70.1 h780b84a_0 conda-forge
glpk 5.0 h445213a_0 conda-forge
gmp 6.2.1 h58526e2_0 conda-forge
googledrivedownloader 0.4 pyhd3deb0d_1 conda-forge
graphite2 1.3.13 h58526e2_1001 conda-forge
gst-plugins-base 1.18.5 hf529b03_2 conda-forge
gstreamer 1.18.5 h9f60fe5_2 conda-forge
harfbuzz 2.9.1 h83ec7ef_1 conda-forge
hdf4 4.2.15 h10796ff_3 conda-forge
hdf5 1.12.1 nompi_h2750804_102 conda-forge
heapdict 1.0.1 py_0 conda-forge
holoviews 1.14.6 pyhd8ed1ab_0 conda-forge
hvplot 0.7.3 pyh6c4a22f_0 conda-forge
icu 68.2 h9c3ff4c_0 conda-forge
identify 2.3.7 pyhd8ed1ab_0 conda-forge
idna 3.1 pyhd3deb0d_0 conda-forge
importlib-metadata 4.8.2 py39hf3d152e_0 conda-forge
importlib_metadata 4.8.2 hd8ed1ab_0 conda-forge
importlib_resources 5.4.0 pyhd8ed1ab_0 conda-forge
iniconfig 1.1.1 pyh9f0ad1d_0 conda-forge
ipopt 3.14.4 h7ede334_0 conda-forge
ipykernel 6.5.0 py39hef51801_1 conda-forge
ipython 7.29.0 py39hef51801_2 conda-forge
ipython_genutils 0.2.0 py_1 conda-forge
jbig 2.1 h7f98852_2003 conda-forge
jedi 0.18.1 py39hf3d152e_0 conda-forge
jinja2 3.0.3 pyhd8ed1ab_0 conda-forge
joblib 1.1.0 pyhd8ed1ab_0 conda-forge
jpeg 9d h36c2ea0_0 conda-forge
json-c 0.15 h98cffda_0 conda-forge
json5 0.9.5 pyh9f0ad1d_0 conda-forge
jsonschema 4.2.1 pyhd8ed1ab_0 conda-forge
jupyter_client 7.1.0 pyhd8ed1ab_0 conda-forge
jupyter_core 4.9.1 py39hf3d152e_1 conda-forge
jupyter_server 1.12.0 pyhd8ed1ab_0 conda-forge
jupyterlab 3.2.4 pyhd8ed1ab_0 conda-forge
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_server 2.8.2 pyhd8ed1ab_0 conda-forge
kealib 1.4.14 h87e4c3c_3 conda-forge
kiwisolver 1.3.2 py39h1a9c180_1 conda-forge
krb5 1.19.2 hcc1bbae_3 conda-forge
lcms2 2.12 hddcbb42_0 conda-forge
ld_impl_linux-64 2.36.1 hea4e1c9_2 conda-forge
lerc 3.0 h9c3ff4c_0 conda-forge
libarchive 3.5.2 hccf745f_1 conda-forge
libblas 3.9.0 12_linux64_openblas conda-forge
libbrotlicommon 1.0.9 h7f98852_6 conda-forge
libbrotlidec 1.0.9 h7f98852_6 conda-forge
libbrotlienc 1.0.9 h7f98852_6 conda-forge
libcblas 3.9.0 12_linux64_openblas conda-forge
libclang 11.1.0 default_ha53f305_1 conda-forge
libcurl 7.80.0 h2574ce0_0 conda-forge
libdap4 3.20.6 hd7c4107_2 conda-forge
libdeflate 1.8 h7f98852_0 conda-forge
libedit 3.1.20191231 he28a2e2_2 conda-forge
libev 4.33 h516909a_1 conda-forge
libevent 2.1.10 h9b69904_4 conda-forge
libffi 3.4.2 h7f98852_5 conda-forge
libgcc-ng 11.2.0 h1d223b6_11 conda-forge
libgdal 3.3.3 h356f897_0 conda-forge
libgfortran-ng 11.2.0 h69a702a_11 conda-forge
libgfortran5 11.2.0 h5c6108e_11 conda-forge
libglib 2.70.1 h174f98d_0 conda-forge
libgomp 11.2.0 h1d223b6_11 conda-forge
libiconv 1.16 h516909a_0 conda-forge
libkml 1.3.0 h238a007_1014 conda-forge
liblapack 3.9.0 12_linux64_openblas conda-forge
libllvm10 10.0.1 he513fc3_3 conda-forge
libllvm11 11.1.0 hf817b99_2 conda-forge
libmamba 0.18.2 h3985d26_0 conda-forge
libmambapy 0.18.2 py39h8bfa403_0 conda-forge
libnetcdf 4.8.1 nompi_hb3fd0d9_101 conda-forge
libnghttp2 1.43.0 h812cca2_1 conda-forge
libnsl 2.0.0 h7f98852_0 conda-forge
libogg 1.3.4 h7f98852_1 conda-forge
libopenblas 0.3.18 pthreads_h8fe5266_0 conda-forge
libopus 1.3.1 h7f98852_1 conda-forge
libpng 1.6.37 h21135ba_2 conda-forge
libpq 13.5 hd57d9b9_0 conda-forge
librttopo 1.1.0 h1185371_6 conda-forge
libsodium 1.0.18 h36c2ea0_1 conda-forge
libsolv 0.7.19 h780b84a_5 conda-forge
libspatialindex 1.9.3 h9c3ff4c_4 conda-forge
libspatialite 5.0.1 h8796b1e_9 conda-forge
libssh2 1.10.0 ha56f1ee_2 conda-forge
libstdcxx-ng 11.2.0 he4da1e4_11 conda-forge
libtiff 4.3.0 h6f004c6_2 conda-forge
libuuid 2.32.1 h7f98852_1000 conda-forge
libvorbis 1.3.7 h9c3ff4c_0 conda-forge
libwebp-base 1.2.1 h7f98852_0 conda-forge
libxcb 1.13 h7f98852_1004 conda-forge
libxkbcommon 1.0.3 he3ba5ed_0 conda-forge
libxml2 2.9.12 h72842e0_0 conda-forge
libxslt 1.1.33 h15afd5d_2 conda-forge
libzip 1.8.0 h4de3113_1 conda-forge
libzlib 1.2.11 h36c2ea0_1013 conda-forge
llvmlite 0.36.0 py39h1bbdace_0 conda-forge
locket 0.2.0 py_2 conda-forge
lxml 4.6.4 py39h107f48f_0 conda-forge
lz4-c 1.9.3 h9c3ff4c_1 conda-forge
lzo 2.10 h516909a_1000 conda-forge
mamba 0.18.2 py39hfa8f2c8_0 conda-forge
mapclassify 2.4.3 pyhd8ed1ab_0 conda-forge
markdown 3.3.6 pyhd8ed1ab_0 conda-forge
markupsafe 2.0.1 py39h3811e60_1 conda-forge
matplotlib 3.5.0 py39hf3d152e_0 conda-forge
matplotlib-base 3.5.0 py39h2fa2bec_0 conda-forge
matplotlib-inline 0.1.3 pyhd8ed1ab_0 conda-forge
memory_profiler 0.58.0 py_0 conda-forge
mercantile 1.2.1 pyhd8ed1ab_0 conda-forge
metis 5.1.0 h58526e2_1006 conda-forge
mistune 0.8.4 py39h3811e60_1005 conda-forge
mock 4.0.3 py39hf3d152e_2 conda-forge
more-itertools 8.12.0 pyhd8ed1ab_0 conda-forge
msgpack-python 1.0.3 py39h1a9c180_0 conda-forge
multipledispatch 0.6.0 py_0 conda-forge
mumps-include 5.2.1 ha770c72_10 conda-forge
mumps-seq 5.2.1 h47a8eb5_10 conda-forge
munch 2.5.0 py_0 conda-forge
munkres 1.1.4 pyh9f0ad1d_0 conda-forge
mysql-common 8.0.27 ha770c72_1 conda-forge
mysql-libs 8.0.27 hfa10184_1 conda-forge
nbclassic 0.3.4 pyhd8ed1ab_0 conda-forge
nbclient 0.5.9 pyhd8ed1ab_0 conda-forge
nbconvert 6.3.0 py39hf3d152e_1 conda-forge
nbformat 5.1.3 pyhd8ed1ab_0 conda-forge
ncurses 6.2 h58526e2_4 conda-forge
nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge
netcdf4 1.5.8 nompi_py39h64b754b_101 conda-forge
networkx 2.6.3 pyhd8ed1ab_1 conda-forge
nodeenv 1.6.0 pyhd8ed1ab_0 conda-forge
nomkl 1.0 h5ca1d4c_0 conda-forge
notebook 6.4.6 pyha770c72_0 conda-forge
nspr 4.32 h9c3ff4c_1 conda-forge
nss 3.72 hb5efdd6_0 conda-forge
numba 0.53.1 py39h56b8d98_1 conda-forge
numexpr 2.7.3 py39hbd72853_102 conda-forge
numpy 1.21.4 py39hdbf815f_0 conda-forge
olefile 0.46 pyh9f0ad1d_1 conda-forge
openjdk 11.0.9.1 h5cc2fde_1 conda-forge
openjpeg 2.4.0 hb52868f_1 conda-forge
openpyxl 3.0.9 pyhd8ed1ab_0 conda-forge
openssl 1.1.1l h7f98852_0 conda-forge
packaging 21.3 pyhd8ed1ab_0 conda-forge
pandas 1.3.4 py39hde0f152_1 conda-forge
pandoc 2.16.2 h7f98852_0 conda-forge
pandocfilters 1.5.0 pyhd8ed1ab_0 conda-forge
panel 0.12.4 pyhd8ed1ab_0 conda-forge
param 1.12.0 pyh6c4a22f_0 conda-forge
parso 0.8.2 pyhd8ed1ab_0 conda-forge
partd 1.2.0 pyhd8ed1ab_0 conda-forge
patsy 0.5.2 pyhd8ed1ab_0 conda-forge
pcre 8.45 h9c3ff4c_0 conda-forge
pexpect 4.8.0 pyh9f0ad1d_2 conda-forge
pickleshare 0.7.5 py_1003 conda-forge
pillow 8.4.0 py39ha612740_0 conda-forge
pip 21.3.1 pyhd8ed1ab_0 conda-forge
pixman 0.40.0 h36c2ea0_0 conda-forge
pluggy 1.0.0 py39hf3d152e_2 conda-forge
ply 3.11 py_1 conda-forge
poppler 21.09.0 ha39eefc_3 conda-forge
poppler-data 0.4.11 hd8ed1ab_0 conda-forge
postgresql 13.5 h2510834_0 conda-forge
powerplantmatching 0.4.8 pyhd8ed1ab_0 conda-forge
pre-commit 2.15.0 py39hf3d152e_1 conda-forge
progressbar2 3.55.0 pyhd8ed1ab_0 conda-forge
proj 8.1.1 h277dcde_2 conda-forge
prometheus_client 0.12.0 pyhd8ed1ab_0 conda-forge
prompt-toolkit 3.0.22 pyha770c72_0 conda-forge
protobuf 3.19.1 pypi_0 pypi
psutil 5.8.0 py39h3811e60_2 conda-forge
pthread-stubs 0.4 h36c2ea0_1001 conda-forge
ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge
pulp 2.5.1 py39hf3d152e_1 conda-forge
py 1.11.0 pyh6c4a22f_0 conda-forge
pybind11-abi 4 hd8ed1ab_3 conda-forge
pycosat 0.6.3 py39h3811e60_1009 conda-forge
pycountry 20.7.3 pyh9f0ad1d_0 conda-forge
pycparser 2.21 pyhd8ed1ab_0 conda-forge
pyct 0.4.6 py_0 conda-forge
pyct-core 0.4.6 py_0 conda-forge
pygments 2.10.0 pyhd8ed1ab_0 conda-forge
pyomo 6.2 py39he80948d_0 conda-forge
pyopenssl 21.0.0 pyhd8ed1ab_0 conda-forge
pyparsing 3.0.6 pyhd8ed1ab_0 conda-forge
pyproj 3.2.1 py39ha81a305_2 conda-forge
pypsa 0.18.1 pyhd8ed1ab_0 conda-forge
pyqt 5.12.3 py39hf3d152e_8 conda-forge
pyqt-impl 5.12.3 py39hde8b62d_8 conda-forge
pyqt5-sip 4.19.18 py39he80948d_8 conda-forge
pyqtchart 5.12 py39h0fcd23e_8 conda-forge
pyqtwebengine 5.12.1 py39h0fcd23e_8 conda-forge
pyrsistent 0.18.0 py39h3811e60_0 conda-forge
pyshp 2.1.3 pyh44b312d_0 conda-forge
pysocks 1.7.1 py39hf3d152e_4 conda-forge
pytables 3.6.1 py39h2669a42_4 conda-forge
pytest 6.2.5 py39hf3d152e_1 conda-forge
python 3.9.7 hb7a2778_3_cpython conda-forge
python-dateutil 2.8.2 pyhd8ed1ab_0 conda-forge
python-utils 2.5.6 pyh44b312d_0 conda-forge
python_abi 3.9 2_cp39 conda-forge
pytz 2021.3 pyhd8ed1ab_0 conda-forge
pyviz_comms 2.1.0 pyhd8ed1ab_0 conda-forge
pyyaml 6.0 py39h3811e60_3 conda-forge
pyzmq 22.3.0 py39h37b5a0c_1 conda-forge
qt 5.12.9 hda022c4_4 conda-forge
rasterio 1.2.10 py39hb37810a_0 conda-forge
ratelimiter 1.2.0 py_1002 conda-forge
readline 8.1 h46c0cb4_0 conda-forge
reproc 14.2.3 h7f98852_0 conda-forge
reproc-cpp 14.2.3 h9c3ff4c_0 conda-forge
requests 2.26.0 pyhd8ed1ab_1 conda-forge
rioxarray 0.9.0 pyhd8ed1ab_0 conda-forge
rtree 0.9.7 py39hb102c33_3 conda-forge
ruamel_yaml 0.15.80 py39h3811e60_1006 conda-forge
scikit-learn 1.0.1 py39h4dfa638_2 conda-forge
scipy 1.7.3 py39hee8e79c_0 conda-forge
scotch 6.0.9 h3858553_1 conda-forge
seaborn 0.11.2 hd8ed1ab_0 conda-forge
seaborn-base 0.11.2 pyhd8ed1ab_0 conda-forge
send2trash 1.8.0 pyhd8ed1ab_0 conda-forge
setuptools 59.2.0 py39hf3d152e_0 conda-forge
shapely 1.8rc1 py39h2226dc9_0 conda-forge/label/shapely_dev
six 1.16.0 pyh6c4a22f_0 conda-forge
sklearn 0.0 pypi_0 pypi
smart_open 5.2.1 pyhd8ed1ab_0 conda-forge
smmap 3.0.5 pyh44b312d_0 conda-forge
snakemake-minimal 6.11.0 pyhdfd78af_0 bioconda
sniffio 1.2.0 py39hf3d152e_2 conda-forge
snuggs 1.4.7 py_0 conda-forge
sortedcontainers 2.4.0 pyhd8ed1ab_0 conda-forge
soupsieve 2.3 pyhd8ed1ab_0 conda-forge
sqlite 3.36.0 h9cd32fc_2 conda-forge
statsmodels 0.13.1 py39hce5d2b2_0 conda-forge
stopit 1.1.2 py_0 conda-forge
tabulate 0.8.9 pyhd8ed1ab_0 conda-forge
tblib 1.7.0 pyhd8ed1ab_0 conda-forge
terminado 0.12.1 py39hf3d152e_1 conda-forge
testpath 0.5.0 pyhd8ed1ab_0 conda-forge
threadpoolctl 3.0.0 pyh8a188c0_0 conda-forge
tiledb 2.3.4 he87e0bf_0 conda-forge
tk 8.6.11 h27826a3_1 conda-forge
toml 0.10.2 pyhd8ed1ab_0 conda-forge
toolz 0.11.2 pyhd8ed1ab_0 conda-forge
toposort 1.7 pyhd8ed1ab_0 conda-forge
tornado 6.1 py39h3811e60_2 conda-forge
tqdm 4.62.3 pyhd8ed1ab_0 conda-forge
traitlets 5.1.1 pyhd8ed1ab_0 conda-forge
tsam 1.1.2 pypi_0 pypi
typing_extensions 4.0.0 pyha770c72_0 conda-forge
tzcode 2021e h7f98852_0 conda-forge
tzdata 2021e he74cb21_0 conda-forge
urllib3 1.26.7 pyhd8ed1ab_0 conda-forge
virtualenv 20.4.7 py39hf3d152e_1 conda-forge
vresutils 0.3.1 pypi_0 pypi
wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge
webencodings 0.5.1 py_1 conda-forge
websocket-client 1.2.1 py39hf3d152e_0 conda-forge
wheel 0.37.0 pyhd8ed1ab_1 conda-forge
wrapt 1.13.3 py39h3811e60_1 conda-forge
xarray 0.20.2.dev19+g21e84842 pypi_0 pypi
xerces-c 3.2.3 h9d8b166_3 conda-forge
xlrd 2.0.1 pyhd8ed1ab_3 conda-forge
xorg-fixesproto 5.0 h7f98852_1002 conda-forge
xorg-inputproto 2.3.2 h7f98852_1002 conda-forge
xorg-kbproto 1.0.7 h7f98852_1002 conda-forge
xorg-libice 1.0.10 h7f98852_0 conda-forge
xorg-libsm 1.2.3 hd9c2040_1000 conda-forge
xorg-libx11 1.7.2 h7f98852_0 conda-forge
xorg-libxau 1.0.9 h7f98852_0 conda-forge
xorg-libxdmcp 1.1.3 h7f98852_0 conda-forge
xorg-libxext 1.3.4 h7f98852_1 conda-forge
xorg-libxfixes 5.0.3 h7f98852_1004 conda-forge
xorg-libxi 1.7.10 h7f98852_0 conda-forge
xorg-libxrender 0.9.10 h7f98852_1003 conda-forge
xorg-libxtst 1.2.3 h7f98852_1002 conda-forge
xorg-recordproto 1.14.2 h7f98852_1002 conda-forge
xorg-renderproto 0.11.1 h7f98852_1002 conda-forge
xorg-xextproto 7.3.0 h7f98852_1002 conda-forge
xorg-xproto 7.0.31 h7f98852_1007 conda-forge
xyzservices 2021.11.0 pyhd8ed1ab_0 conda-forge
xz 5.2.5 h516909a_1 conda-forge
yaml 0.2.5 h516909a_0 conda-forge
yaml-cpp 0.6.3 he1b5a44_4 conda-forge
zeromq 4.3.4 h9c3ff4c_1 conda-forge
zict 2.0.0 py_0 conda-forge
zipp 3.6.0 pyhd8ed1ab_0 conda-forge
zlib 1.2.11 h36c2ea0_1013 conda-forge
zstd 1.5.0 ha95c52a_0 conda-forge
```

[euronion]

Can reproduce. From a first glance this might be an issue on the CDS API side. classic netCDF is limited in size to ~2 GB, maybe the individual requests are exceeding this size and the api does not internally switch to netCDF v4 (which allows for larger sizes)? We'd need to investigate a bit more, maybe someone else has a clearer view?

Full stack trace:

2022-02-22 08:15:33,413 INFO Requesting data for feature influx...
2022-02-22 08:23:52,061 INFO CDS: Downloading variables
	 * geopotential (2013)

2022-02-22 08:59:55,175 ERROR Message: the request you have submitted is not valid                                                                                                                                                                                                                                                                   
2022-02-22 08:59:55,176 ERROR Reason:  
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.

2022-02-22 08:59:55,177 ERROR   Traceback (most recent call last):
2022-02-22 08:59:55,178 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/cdshandlers/services/handler.py", line 55, in handle_request
2022-02-22 08:59:55,179 ERROR       result = cached(context.method, proc, context, context.args, context.kwargs)
2022-02-22 08:59:55,179 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/caching.py", line 108, in cached
2022-02-22 08:59:55,180 ERROR       result = proc(context, *context.args, **context.kwargs)
2022-02-22 08:59:55,180 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 118, in __call__
2022-02-22 08:59:55,180 ERROR       return p(*args, **kwargs)
2022-02-22 08:59:55,181 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/services.py", line 59, in __call__
2022-02-22 08:59:55,181 ERROR       return self.proc(context, *args, **kwargs)
2022-02-22 08:59:55,181 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 47, in internal
2022-02-22 08:59:55,181 ERROR       return mars(context, request, **kwargs)
2022-02-22 08:59:55,182 ERROR     File "/home/cds/cdsservices/services/mars/mars.py", line 25, in mars
2022-02-22 08:59:55,182 ERROR       grib_to_netcdf(context, requests, info)
2022-02-22 08:59:55,182 ERROR     File "/home/cds/cdsservices/services/mars/grib_to_netcdf.py", line 42, in grib_to_netcdf
2022-02-22 08:59:55,183 ERROR       context.run_command(*cmd, exception=NetcdfException)
2022-02-22 08:59:55,183 ERROR     File "/opt/cdstoolbox/cdscompute/cdscompute/context.py", line 209, in run_command
2022-02-22 08:59:55,184 ERROR       raise exception(call, proc.returncode, output)
2022-02-22 08:59:55,184 ERROR   home.cds.cdsservices.services.mars.__init__.py.exceptions.NetcdfException: 
2022-02-22 08:59:55,185 ERROR   grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

---------------------------------------------------------------------------
Exception                                 Traceback (most recent call last)
Input In [6], in <module>
----> 1 cutout.prepare()

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:102, in maybe_remove_tmpdir.<locals>.wrapper(*args, **kwargs)
    100 kwargs["tmpdir"] = mkdtemp()
    101 try:
--> 102     res = func(*args, **kwargs)
    103 finally:
    104     rmtree(kwargs["tmpdir"])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:164, in cutout_prepare(cutout, features, tmpdir, overwrite)
    162 logger.info(f"Calculating and writing with module {module}:")
    163 missing_features = missing_vars.index.unique("feature")
--> 164 ds = get_features(cutout, module, missing_features, tmpdir=tmpdir)
    165 prepared |= set(missing_features)
    167 cutout.data.attrs.update(dict(prepared_features=list(prepared)))

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/data.py:46, in get_features(cutout, module, features, tmpdir)
     41     feature_data = delayed(get_data)(
     42         cutout, feature, tmpdir=tmpdir, lock=lock, **parameters
     43     )
     44     datasets.append(feature_data)
---> 46 datasets = compute(*datasets)
     48 ds = xr.merge(datasets, compat="equals")
     49 for v in ds:

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/base.py:571, in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
    568     keys.append(x.__dask_keys__())
    569     postcomputes.append(x.__dask_postcompute__())
--> 571 results = schedule(dsk, keys, **kwargs)
    572 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/threaded.py:79, in get(dsk, result, cache, num_workers, pool, **kwargs)
     76     elif isinstance(pool, multiprocessing.pool.Pool):
     77         pool = MultiprocessingPoolExecutor(pool)
---> 79 results = get_async(
     80     pool.submit,
     81     pool._max_workers,
     82     dsk,
     83     result,
     84     cache=cache,
     85     get_id=_thread_get_id,
     86     pack_exception=pack_exception,
     87     **kwargs,
     88 )
     90 # Cleanup pools associated to dead threads
     91 with pools_lock:

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:507, in get_async(submit, num_workers, dsk, result, cache, get_id, rerun_exceptions_locally, pack_exception, raise_exception, callbacks, dumps, loads, chunksize, **kwargs)
    505         _execute_task(task, data)  # Re-execute locally
    506     else:
--> 507         raise_exception(exc, tb)
    508 res, worker_id = loads(res_info)
    509 state["cache"][key] = res

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:315, in reraise(exc, tb)
    313 if exc.__traceback__ is not tb:
    314     raise exc.with_traceback(tb)
--> 315 raise exc

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/local.py:220, in execute_task(key, task_info, dumps, loads, get_id, pack_exception)
    218 try:
    219     task, data = loads(task_info)
--> 220     result = _execute_task(task, data)
    221     id = get_id()
    222     result = dumps((result, id))

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/core.py:119, in _execute_task(arg, cache, dsk)
    115     func, args = arg[0], arg[1:]
    116     # Note: Don't assign the subtask results to a variable. numpy detects
    117     # temporaries by their reference count and can execute certain
    118     # operations in-place.
--> 119     return func(*(_execute_task(a, cache) for a in args))
    120 elif not ishashable(arg):
    121     return arg

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/dask/utils.py:40, in apply(func, args, kwargs)
     38 def apply(func, args, kwargs=None):
     39     if kwargs:
---> 40         return func(*args, **kwargs)
     41     else:
     42         return func(*args)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:378, in get_data(cutout, feature, tmpdir, lock, **creation_parameters)
    374     return retrieve_once(retrieval_times(coords, static=True)).squeeze()
    376 datasets = map(retrieve_once, retrieval_times(coords))
--> 378 return xr.concat(datasets, dim="time").sel(time=coords["time"])

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/concat.py:220, in concat(objs, dim, data_vars, coords, compat, positions, fill_value, join, combine_attrs)
    217 from .dataset import Dataset
    219 try:
--> 220     first_obj, objs = utils.peek_at(objs)
    221 except StopIteration:
    222     raise ValueError("must supply at least one object to concatenate")

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/xarray/core/utils.py:196, in peek_at(iterable)
    192 """Returns the first value from iterable, as well as a new iterator with
    193 the same content as the original iterable
    194 """
    195 gen = iter(iterable)
--> 196 peek = next(gen)
    197 return peek, itertools.chain([peek], gen)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:368, in get_data.<locals>.retrieve_once(time)
    367 def retrieve_once(time):
--> 368     ds = func({**retrieval_params, **time})
    369     if sanitize and sanitize_func is not None:
    370         ds = sanitize_func(ds)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:192, in get_data_temperature(retrieval_params)
    190 def get_data_temperature(retrieval_params):
    191     """Get wind temperature for given retrieval parameters."""
--> 192     ds = retrieve_data(
    193         variable=["2m_temperature", "soil_temperature_level_4"], **retrieval_params
    194     )
    196     ds = _rename_and_clean_coords(ds)
    197     ds = ds.rename({"t2m": "temperature", "stl4": "soil temperature"})

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/atlite/datasets/era5.py:301, in retrieve_data(product, chunks, tmpdir, lock, **updates)
    294 assert {"year", "month", "variable"}.issubset(
    295     request
    296 ), "Need to specify at least 'variable', 'year' and 'month'"
    298 client = cdsapi.Client(
    299     info_callback=logger.debug, debug=logging.DEBUG >= logging.root.level
    300 )
--> 301 result = client.retrieve(product, request)
    303 if lock is None:
    304     lock = nullcontext()

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:348, in Client.retrieve(self, name, request, target)
    347 def retrieve(self, name, request, target=None):
--> 348     result = self._api("%s/resources/%s" % (self.url, name), request, "POST")
    349     if target is not None:
    350         result.download(target)

File ~/miniconda3/envs/atlite/lib/python3.8/site-packages/cdsapi/api.py:506, in Client._api(self, url, request, method)
    504             break
    505         self.error("  %s", n)
--> 506     raise Exception(
    507         "%s. %s."
    508         % (reply["error"].get("message"), reply["error"].get("reason"))
    509     )
    511 raise Exception("Unknown API state [%s]" % (reply["state"],))

Exception: the request you have submitted is not valid. 
grib_to_netcdf ERROR: line 4334, nc_enddef: NetCDF: One or more variable sizes violate format constraints

Cannot create netCDF classic format, dataset is too large!
Try splitting the input GRIB(s).
grib_to_netcdf: Version 2.24.2
grib_to_netcdf: Processing input file '/cache/tmp/74fcce81-8102-4388-b11b-f71e8c4d56a3-adaptor.mars.internal-1645514525.1224709-4276-8-tmp.grib'.
grib_to_netcdf: Found 17520 GRIB fields in 1 file.
grib_to_netcdf: Ignoring key(s): method, type, stream, refdate, hdate
grib_to_netcdf: Creating netCDF file '/cache/data5/adaptor.mars.internal-1645515842.3469903-4276-6-74fcce81-8102-4388-b11b-f71e8c4d56a3.nc'
grib_to_netcdf: NetCDF library version: 4.3.3.1 of Dec 10 2015 16:44:18 $
grib_to_netcdf: Creating large (64 bit) file format.
grib_to_netcdf: Defining variable 't2m'.
grib_to_netcdf: Defining variable 'stl4'.
.

[davide-f]

I'm wondering whether atlite needs to download copernicus data as NC format; the nc extraction is also experimental in copernicus.
A possible solution could be to use the raw grib format when downloading data from copernicus, xarray should be able to load such data accordingly, yet I'm not sure whether the loading may need some adaptations.

https://stackoverflow.com/questions/64937550/converting-grib-to-netcdf4

Update: maybe not possible; see https://confluence.ecmwf.int/display/CKB/Climate+Data+Store+%28CDS%29+API+Keywords