Use more recent curl version to fix hydro basins download

[jnnr]

Fixes #267.

Checklist

• CHANGELOG updated
• Minimal workflow tests pass (on jnnr's linux machine)
• Tests added to cover contribution
• Documentation updated
• Configuration schema updated

[timtroendle]

Awesome! Could you also update the CHANGELOG.md? This seems to fit into the ### Fixed (workflow) section.

[jnnr]

Thanks for your review, @timtroendle! The minimal workflow is still running. I'll check the box if it finishes successfully.

[jnnr]

Unfortunately, when running the minimal workflow I get a new error:

Error in rule download_raw_load:
    jobid: 75
    output: data/automatic/raw-load-data.csv
    conda-env: /home/jlauner/repos/envs/snakemake/euro-calliope/eaf9c112ec3dcd126c6a04c49f3548b5_
    shell:
        curl -sLo data/automatic/raw-load-data.csv 'https://data.open-power-system-data.org/time_series/2019-06-05/time_series_60min_stacked.csv'
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

and

Error in rule download_capacity_factors_wind_and_solar:
    jobid: 53
    output: data/automatic/capacityfactors/wind-onshore-timeseries.nc
    conda-env: /home/jlauner/repos/envs/snakemake/euro-calliope/eaf9c112ec3dcd126c6a04c49f3548b5_
    shell:
        curl -sLo data/automatic/capacityfactors/wind-onshore-timeseries.nc 'https://zenodo.org/record/3899687/files/wind-onshore-timeseries.nc?download=1'
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

[timtroendle]

Oh no, how annoying. The whole point of providing curl through Snakemake was to INCREASE portability, not DECREASE it. See #64. Seems we failed.

I finally managed to reproduce the original problem with version 7.76.0. It indeed does fail on our Linux system, too! Seems we haven't run the download for quite a while.

Unfortunately, I again fail to reproduce the problem with version 8.6.0. All downloads (basins, load, capacity-factors) work just fine using version 8.6.0, build hca28451_0, from channel conda-forge/linux-64. Could you check if your package deviates from this?

Also, I figured why you don't see error messages: curl requires the S option in addition to the s option to show error message. Regardless of the solution here, we should add the S option to all uses of curl in the repository.

Could you rerun with the S option and post the error message?

[timtroendle]

They have also released version 8.7.1 just a few days ago. In the absence of other solutions, you could also try updating again.

[sjpfenninger]

I have just re-run again after wiping all build and data files to ensure everything is downloaded, and for me, it works with curl 8.6.0 (hca28451_0).

I do have another issue that I haven't raised so far but that I always had to fix to get the shell environment working to begin with (on linux only, #321).

[jnnr]

I ran this again on my own computer and esm1. On my own computer, at first run I encountered the above errors again when downloading data/automatic/capacityfactors/wind-onshore-timeseries.nc and data/automatic/raw-load-data.csv. After starting the workflow again it succeeded however without me conciously changing anything.

So I'd suggest we count this as resolved and open a new issue if problems persist.