load_earth_relief: Add the support of data source "GEBCO" #1818

willschlitzer · 2022-03-15T10:04:02Z

This pull request adds the GEBCO data source to load_earth_relief

Addresses #1786

Reminders

Run make format and make check to make sure the code follows the style guide.
Add tests for new features or tests that would have caught the bug that you're fixing.
Add new public functions/methods/classes to doc/api/index.rst.
Write detailed docstrings for all functions/methods.
If wrapping a new module, open a 'Wrap new GMT module' issue and submit reasonably-sized PRs.
If adding new functionality, add an example to docstrings or tutorials.

Slash Commands

You can write slash commands (/command) in the first line of a comment to perform
specific operations. Supported slash commands are:

/format: automatically format and lint the code
/test-gmt-dev: run full tests on the latest GMT development version

willschlitzer · 2022-03-15T10:12:02Z

I'm running into an error when my machine is trying to find the cached version of the GEBCO files. When I run both grid = load_earth_relief(data_source="gebco", region = [0, 5, 10, 12]) and grid = pygmt.grdcut(grid="@earth_gebco_01d", region = [0, 5, 10, 12]) I get the following:

GMTCLibError: Module 'grdcut' failed with status code 16:
grdcut [ERROR]: Libcurl Error: HTTP response code said error
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd not found

My guess is the issue is that the Earth GEBCO files are stored under .gmt/server/earth/earth_gebco rather than .gmt/cache. Is this a GMT error for how it finds the cached files?

seisman · 2022-03-15T11:26:50Z

What's the output if you run the following GMT command?

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd

willschlitzer · 2022-03-15T12:18:55Z

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd

Same issue. Does it work for you?

Here is the output:

grdcut [NOTICE]:   -> Download grid file [105K]: earth_gebco_01d_g.grd
grdcut [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/cache/earth_gebco_01d.grd ...
grdcut [ERROR]: Libcurl Error: HTTP response code said error
grdcut [WARNING]: You can turn remote file download off by setting GMT_DATA_UPDATE_INTERVAL to "off"
grdcut [DEBUG]: Delete /tmp/earth_gebco_01d.grd.download
grdcut [DEBUG]: Replace file @earth_gebco_01d.grd with /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [INFORMATION]: Processing input grid
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd was not found
grdcut [ERROR]: Cannot find file /home/will/.gmt/cache/earth_gebco_01d.grd
grdcut [ERROR]: File /home/will/.gmt/cache/earth_gebco_01d.grd not found
[Session gmt (0)]: Error returned from GMT API: GMT_FILE_NOT_FOUND (16)

The file earth_gebco_01d_g.grd is in .gmt/server/earth/earth_gebco, so my guess is that GMT is looking in the wrong folder for the .grd file?

weiji14 · 2022-03-15T13:35:57Z

I've got the same GMT_FILE_NOT_FOUND error when using grdcut, but using gmt which -Ga @earth_gebco_01d_g works:

gmt [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/gmt_data_server.txt ...
gmt [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/gmt_hash_server.txt ...
gmtwhich [INFORMATION]: Writing Data Table to Standard Output stream
gmtwhich [NOTICE]: Remote data courtesy of GMT data server oceania [http://oceania.generic-mapping-tools.org]

gmtwhich [NOTICE]: GEBCO Earth Relief at 1x1 arc degrees reduced by Gaussian Cartesian filtering (111.2 km fullwidth) [GEBCO Compilation Group, 2021 GEBCO 2021 Grid].
gmtwhich [NOTICE]:   -> Download grid file [105K]: earth_gebco_01d_g.grd
gmtwhich [INFORMATION]: Downloading file http://oceania.generic-mapping-tools.org/server/earth/earth_gebco/earth_gebco_01d_g.grd ...
/home/username/.gmt/server/earth/earth_gebco/earth_gebco_01d_g.grd

willschlitzer · 2022-03-15T15:17:02Z

@weiji14 Does the use of which request a download from the remote server regardless or if it's cached, or will it use the cached version of the file if available? I was able to download the .grd and .nc files to .gmt/server/earth/earth_gebco.

weiji14 · 2022-03-15T17:23:05Z

@weiji14 Does the use of which request a download from the remote server regardless or if it's cached, or will it use the cached version of the file if available? I was able to download the .grd and .nc files to .gmt/server/earth/earth_gebco.

It'll definitely take the local cached file. Looking at the verbose log though, which does seem to ping server network first to check for a new gmt_data_server.txt file and will pull an update if needed.

maxrjones · 2022-03-15T19:40:10Z

These issues could be related to GenericMappingTools/gmt#6160. I would be curious if it works after deleting ~/.gmt/server/gmt_data_server.txt

willschlitzer · 2022-03-16T08:08:32Z

These issues could be related to GenericMappingTools/gmt#6160. I would be curious if it works after deleting ~/.gmt/server/gmt_data_server.txt

Just deleted both gmt_data_server.txt and gmt_data_server,tx.old; same result

maxrjones · 2022-03-18T18:40:16Z

gmt grdcut @earth_gebco_01d -R0/5/10/20 -Gout.nc -Vd fails for me with GMT 6.3.0 but works with the main branch, due to the fix applied in GenericMappingTools/gmt#6160. We may want to wait until after the GMT 6.4.0 release to add support for earth_gebco and earth_geoid.

willschlitzer · 2022-09-13T14:29:36Z

How extensively should this be testing load_earth_relief with data_source="gebco"? I don't want to duplicate all of the tests for Earth relief using SRTM data, as that seems excessive, so I just added tests using 1-degree resolution with and without setting region.

seisman · 2022-09-14T05:47:09Z

How extensively should this be testing load_earth_relief with data_source="gebco"? I don't want to duplicate all of the tests for Earth relief using SRTM data, as that seems excessive, so I just added tests using 1-degree resolution with and without setting region.

Yes, I agree.

pygmt/datasets/earth_relief.py

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

weiji14 · 2022-09-19T13:45:04Z

pygmt/datasets/earth_relief.py

@@ -55,9 +61,19 @@ def load_earth_relief(resolution="01d", region=None, registration=None, use_srtm
    use_srtm : bool
        By default, the land-only SRTM tiles from NASA are used to generate the
        ``"03s"`` and ``"01s"`` grids, and the missing ocean values are filled
-        by up-sampling the SRTM15+ tiles which have a resolution of 15
+        by up-sampling the SRTM15+V2.1 tiles which have a resolution of 15


Should be v2.4 now according to https://web.archive.org/web/20220809113153/https://www.generic-mapping-tools.org/remote-datasets/earth-relief.html

Suggested change

by up-sampling the SRTM15+V2.1 tiles which have a resolution of 15

by up-sampling the SRTM15+V2.4 tiles which have a resolution of 15

I think it would be better to update "Refer to https://www.generic-mapping-tools.org/remote-datasets/earth-relief.html for more details." above to "Refer to https://www.generic-mapping-tools.org/remote-datasets/index.html for more details about available datasets, including version information and references." so that we don't need to update this on each SRTM15+ release and the link provided is more generic.

pygmt/datasets/earth_relief.py

weiji14 · 2022-10-07T02:34:03Z

Need to add @earth_gebco_01d to the cache too at pygmt/helpers/testing.py right?

seisman · 2022-10-12T05:24:48Z

@GenericMappingTools/pygmt-maintainers Can someone else give this PR a final review before we merge it?

pygmt/datasets/earth_relief.py

pygmt/helpers/testing.py

pygmt/tests/test_datasets_earth_relief.py

Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

…es' into load-earth-relief/multiple-sources

weiji14

Cool, thanks @willschlitzer!

Co-authored-by: Dongdong Tian <seisman.info@gmail.com> Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

willschlitzer added 2 commits March 15, 2022 09:07

add GEBCO option for data_source

2916236

add parametrize to test_earth_relief_fails

80c389f

seisman added this to the 0.7.0 milestone Mar 18, 2022

Merge branch 'main' into load-earth-relief/multiple-sources

0d00c3d

vercel bot temporarily deployed to Preview June 13, 2022 09:02 Inactive

seisman modified the milestones: 0.7.0, 0.8.0 Jun 19, 2022

willschlitzer added 2 commits August 3, 2022 15:55

Merge branch 'main' into load-earth-relief/multiple-sources

66ee213

Merge branch 'main' into load-earth-relief/multiple-sources

11310e3

willschlitzer marked this pull request as ready for review September 13, 2022 13:21

willschlitzer marked this pull request as draft September 13, 2022 13:22

willschlitzer added 2 commits September 13, 2022 09:42

add tests at 01d resolution with and without region set for GEBCO data

db7ed8b

add check and testing for invalid data source

d21df89

willschlitzer marked this pull request as ready for review September 13, 2022 14:19

seisman added the enhancement Improving an existing feature label Sep 14, 2022

seisman reviewed Sep 19, 2022

View reviewed changes

pygmt/datasets/earth_relief.py Outdated Show resolved Hide resolved

pygmt/datasets/earth_relief.py Outdated Show resolved Hide resolved

pygmt/datasets/earth_relief.py Outdated Show resolved Hide resolved

pygmt/datasets/earth_relief.py Outdated Show resolved Hide resolved

willschlitzer and others added 2 commits September 19, 2022 09:18

Apply suggestions from code review

5de32f3

Co-authored-by: Dongdong Tian <seisman.info@gmail.com>

Merge branch 'main' into load-earth-relief/multiple-sources

1b88bf2

weiji14 reviewed Sep 19, 2022

View reviewed changes

willschlitzer added 2 commits October 4, 2022 14:24

wording change in docstring

921dd72

Merge branch 'main' into load-earth-relief/multiple-sources

90374b8

weiji14 added feature Brand new feature and removed enhancement Improving an existing feature labels Oct 7, 2022

willschlitzer added 5 commits October 9, 2022 14:56

add 1d GEBCO dataset to testing.py

cc0dd5f

uncomment "pull_request" in cache_data.yaml to update GitHub cache

94e0aaf

disable pylint on too-many-branches in earth_relief.py

dffbbe1

re-comment "pull_request" in cache_data.yaml

eb86bcf

Merge branch 'main' into load-earth-relief/multiple-sources

e12c227

seisman approved these changes Oct 10, 2022

View reviewed changes

seisman added the final review call This PR requires final review and approval from a second reviewer label Oct 10, 2022

weiji14 reviewed Oct 12, 2022

View reviewed changes

willschlitzer and others added 8 commits October 12, 2022 17:20

Apply suggestions from code review

77831ac

Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

move gebco file name for alaphbetizing

0567f63

move up else statement

0862cf2

change data source check

794d69b

add error handling for use_srtm

fe4ce30

Merge remote-tracking branch 'origin/load-earth-relief/multiple-sourc…

0238d59

…es' into load-earth-relief/multiple-sources

Merge branch 'main' into load-earth-relief/multiple-sources

6f0a7d7

change elif statement to else statement, use dict.get

6fa396b

weiji14 approved these changes Oct 14, 2022

View reviewed changes

seisman merged commit f1af459 into main Oct 14, 2022

seisman deleted the load-earth-relief/multiple-sources branch October 14, 2022 04:38

seisman removed the final review call This PR requires final review and approval from a second reviewer label Oct 14, 2022

weiji14 mentioned this pull request Nov 24, 2022

More GMT data sources in load_earth_relief and other load_earth functions #1786

Closed

10 tasks

seisman changed the title ~~Add GEBCO data source to load_earth_relief~~ load_earth_relief: Add the support of data source "GEBCO" Nov 30, 2022

sixy6e pushed a commit to sixy6e/pygmt that referenced this pull request Dec 21, 2022

Add GEBCO data source to load_earth_relief (GenericMappingTools#1818)

51550b2

Co-authored-by: Dongdong Tian <seisman.info@gmail.com> Co-authored-by: Wei Ji <23487320+weiji14@users.noreply.github.com>

weiji14 mentioned this pull request Aug 22, 2023

Fix tests for Aug 2023 updated remote datasets #2636

Merged

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

load_earth_relief: Add the support of data source "GEBCO" #1818

load_earth_relief: Add the support of data source "GEBCO" #1818

willschlitzer commented Mar 15, 2022

willschlitzer commented Mar 15, 2022

seisman commented Mar 15, 2022

willschlitzer commented Mar 15, 2022 •

edited

Loading

weiji14 commented Mar 15, 2022

willschlitzer commented Mar 15, 2022 •

edited

Loading

weiji14 commented Mar 15, 2022

maxrjones commented Mar 15, 2022

willschlitzer commented Mar 16, 2022

maxrjones commented Mar 18, 2022

willschlitzer commented Sep 13, 2022

seisman commented Sep 14, 2022

weiji14 Sep 19, 2022

maxrjones Sep 19, 2022

weiji14 commented Oct 7, 2022 •

edited

Loading

seisman commented Oct 12, 2022

weiji14 left a comment

	by up-sampling the SRTM15+V2.1 tiles which have a resolution of 15
	by up-sampling the SRTM15+V2.4 tiles which have a resolution of 15

load_earth_relief: Add the support of data source "GEBCO" #1818

load_earth_relief: Add the support of data source "GEBCO" #1818

Conversation

willschlitzer commented Mar 15, 2022

willschlitzer commented Mar 15, 2022

seisman commented Mar 15, 2022

willschlitzer commented Mar 15, 2022 • edited Loading

weiji14 commented Mar 15, 2022

willschlitzer commented Mar 15, 2022 • edited Loading

weiji14 commented Mar 15, 2022

maxrjones commented Mar 15, 2022

willschlitzer commented Mar 16, 2022

maxrjones commented Mar 18, 2022

willschlitzer commented Sep 13, 2022

seisman commented Sep 14, 2022

weiji14 Sep 19, 2022

Choose a reason for hiding this comment

maxrjones Sep 19, 2022

Choose a reason for hiding this comment

weiji14 commented Oct 7, 2022 • edited Loading

seisman commented Oct 12, 2022

weiji14 left a comment

Choose a reason for hiding this comment

willschlitzer commented Mar 15, 2022 •

edited

Loading

willschlitzer commented Mar 15, 2022 •

edited

Loading

weiji14 commented Oct 7, 2022 •

edited

Loading