Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

convert_grib2_to_nc.py: various small issues and intermittent bugs #22

Closed
tlvu opened this issue Dec 17, 2020 · 3 comments
Closed

convert_grib2_to_nc.py: various small issues and intermittent bugs #22

tlvu opened this issue Dec 17, 2020 · 3 comments
Assignees
@tlogan2000

Comments

@tlvu
Copy link
Collaborator

tlvu commented Dec 17, 2020

Script convert_grib2_to_nc.py is working fine. All the following are minor issues. Found while working on bird-house/birdhouse-deploy-ouranos#2 and #21.

1- The conversion from .grib2 to .nc is too verbose for automation purposes. Have these in the logs:

                                   ] | 1% Completed |  1.2s^M[                                        ] | 2% Completed |  1.3s^M[                                        ] | 2% Completed |  1.4s^M[                                        ] | 2% Completed |  1.5s^M[#                                       ] | 2% Completed |  1.6s^M[#                                       ] | 2% Completed |  1.7s^M[#                                       ] | 2% Completed |  1.8s^M[#                                       ] | 2% Completed |  1.9s^M[#                                       ] | 3% Completed |  2.0s^M[#                                       ] | 3% Completed |  2.1s^M[#                                       ] | 3% Completed |  2.2s^M[#                                       ] | 3% Completed |  2.3s^M[#

Need to disable the progress update.

2- Repeated "combining variables and timesteps" for past data. I though we are only interested in the latest data pointed by GEPS_latest.nc

pavics-vdb/ECCC-datamart_sync/convert_grib2_to_nc.py

Lines 121 to 129 in 8219f52

## update symlink recent forecast
symlink = jobs[j]['threddspath'].joinpath('GEPS_latest.nc')
latest = sorted([ll for ll in jobs[j]['threddspath'].glob('*.nc') if symlink.name not in ll.name])[-1]
latest_date = latest.name.split(jobs[j]['pattern'][0])[-1].split('_allP')[0]
# create symlink
symlink.unlink(missing_ok=True) # Delete first
os.chdir(symlink.parent)
os.symlink(latest.name, symlink.name)
since the .ncml

pavics-vdb/1-Datasets/forecasts/eccc_geps/GEPS_latest.ncml

Lines 1 to 2 in 8219f52

<?xml version="1.0" encoding="utf-8"?>
<netcdf xmlns="http://www.unidata.ucar.edu/namespaces/netcdf/ncml-2.2" location="/pavics-data/eccc/forecasts/geps/GEPS_latest.nc"></netcdf>
only reference that GEPS_latest.nc

Downloading grib2 files : attempt 1 of 3
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 12
Done. Found 0 new files
Downloading grib2 files : attempt 2 of 3
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 12
Done. Found 0 new files
Downloading grib2 files : attempt 3 of 3
Checking for updated GEPS files : Forecast 20201217 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201216 12
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 00
Done. Found 0 new files
Checking for updated GEPS files : Forecast 20201215 12
Done. Found 0 new files
coverting to netcdf ....
done coverting  GEPS . It took :  99.2399468421936 seconds
2020121400 : combining variables and timesteps ...
2020121412 : combining variables and timesteps ...
2020121500 : combining variables and timesteps ...
2020121512 : combining variables and timesteps ...
2020121600 : combining variables and timesteps ...
2020121612 : combining variables and timesteps ...

3- Intermittent error in "combining variables and timesteps".

Not always, sometime for the same date, the step "combining variables and timesteps" would work for the previous run, but blow up on the next run and work again after that.

2020121400 : combining variables and timesteps ...
CMC_geps-raw_latlon0p5x0p5_2020121400_allP_allmbrs.nc
[######                                  ] | 15% Completed |  5min 23.8s2020121412 : combining variables and timesteps ...
CMC_geps-raw_latlon0p5x0p5_2020121412_allP_allmbrs.nc
[#######                                 ] | 19% Completed |  4min  4.8s2020121500 : combining variables and timesteps ...
CMC_geps-raw_latlon0p5x0p5_2020121500_allP_allmbrs.nc
Traceback (most recent call last):
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 199, in _acquire_with_cache_info
    file = self._cache[self._key]
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/lru_cache.py", line 53, in __getitem__
    value = self._cache[key]
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/data/tmp/geps_forecast/netcdf/CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121500_P336_allmbrs.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False))]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 305, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 126, in main
    reformat_nc((ncfiles, outfile, jobs[j]['variables']))
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 206, in reformat_nc
    dstmp = xr.open_mfdataset(sorted(ncfiles[v]), combine='nested',
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/api.py", line 948, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/api.py", line 948, in <listcomp>
    datasets = [open_(p, **open_kwargs) for p in paths]
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/api.py", line 572, in open_dataset
    store = opener(filename_or_obj, **extra_kwargs, **backend_kwargs)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 364, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 314, in __init__
    self.format = self.ds.data_model
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 373, in ds
    return self._acquire()
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/netCDF4_.py", line 367, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/opt/conda/envs/birdy/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 187, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
  File "/opt/conda/envs/birdy/lib/python3.8/site-packages/xarray/backends/file_manager.py", line 205, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "netCDF4/_netCDF4.pyx", line 2357, in netCDF4._netCDF4.Dataset.__init__
  File "netCDF4/_netCDF4.pyx", line 1925, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -101] NetCDF: HDF error: b'/data/tmp/geps_forecast/netcdf/CMC_geps-raw_TMP_TGL_2m_latlon0p5x0p5_2020121500_P336_allmbrs.nc'
@tlvu
Copy link
Collaborator Author

tlvu commented Dec 17, 2020

Humm, I am not sure about point 2 and 3 anymore. I've run convert_grib2_to_nc.py so many times manually and unattended in a cronjob that I lost track of which error is from which run.

I might have mixed up the results of different runs and so I am not sure they are real issues or reproducible anymore.

@tlvu
Copy link
Collaborator Author

tlvu commented Dec 17, 2020

Another wierd error in the logs:

  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 307, in <module>
    main()
  File "/data/deploy_data_cache/retrieve_geps_forecasts/pavics-vdb-for-geps-forecasts/ECCC-datamart_sync/convert_grib2_to_nc.py", line 132, in main
    latest = sorted([ll for ll in jobs[j]['threddspath'].glob('*.nc') if symlink.name not in ll.name])[-1]
IndexError: list index out of range

Only on my test VM, not on Medus.

@tlvu
Copy link
Collaborator Author

tlvu commented Dec 18, 2020

Issue 3 and 4 were only reproducible on my perso VM which has only 2 CPU and 8G of ram. After talking to @tlogan2000 it's probably the tiny amount of ram that was the cause. Medus never has this problem so it will probably never happen on Boreas as well.

Issue 2 was resolved in commit a6e4c2b

Issue 1 was resolved in commit a64a444

So all issues are accounted for, closing this issue.

@tlvu tlvu closed this as completed Dec 18, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@tlogan2000 tlogan2000

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@tlvu @tlogan2000