Mirror the netcdf files from a tds/thredds catalog to local disk. Runs locally or in docker.
This is a python3 application
The dependencies are listed in requirements.txt file and can normally be installed using pip
pip install -r requirements.txt
Files will be downloaded from the thredds server into the output folder, which must exist already. NB: any netcdf .nc files which are in the folder but not listed in the catalog will be deleted.
In this example, the files are saved into the /my/output/folder directory with a http politeness delay of 1 second between http requests.
python3 thredds-netcdf-mirror.py --thredds http://thredds.marine.ie/thredds --catalog IMI_ROMS_HYDRO/CONNEMARA_250M_20L_1H/FORECAST --output /my/output/folder --delay 1
When running in docker the typical usage is to map a volume for the output data.
docker build -t thredds-netcdf-mirror .
Here we run the application in docker, saving the files into a volume mapped to /my/output/folder directory with politeness of only 1 second delay between http requests.
docker run --rm -i -t -v /my/output/folder:/output thredds-netcdf-mirror --thredds http://thredds.marine.ie/thredds --catalog IMI_ROMS_HYDRO/CONNEMARA_250M_20L_1H/FORECAST --output /output --delay 1
docker build -t 127.0.0.1:5000/thredds-netcdf-mirror .
docker push 127.0.0.1:5000/thredds-netcdf-mirror
docker run --rm -i -t -v /opt/thredds/connemara_his:/output 127.0.0.1:5000/thredds-netcdf-mirror --thredds http://thredds.marine.ie/thredds --catalog IMI_ROMS_HYDRO/CONNEMARA_250M_20L_1H/FORECAST --output /output --delay 1
- build a smaller docker image by starting from python:3.7-apline (tried and failed)
- schedule in swarm using https://github.com/mcuadros/ofelia