Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable writing of 3-D netCDF data cubes from greenspline and grdinterpolate #4581

Merged
merged 53 commits into from
Dec 29, 2020

Conversation

PaulWessel
Copy link
Member

@PaulWessel PaulWessel commented Dec 16, 2020

Description of proposed changes

This PR completes the gmt_write_nc_cube coding to enable the writing of 3-D layers (with layers an unlimited dimension). It is WIP because I want to complete what GMT_Read_Data with CONTAINER_ONLY should do for cubes, and update gspline_4.sh to no longer write lots of 2-D grids.

Related question whose solution can be separate PR is:

  • Do we add -Q for "cube" options to grd2xyz and xyz2grd so they can handle 4-D data or do we clone and write cube2xyzw/xyzw2cube? My vote would be for -Q since adding two new modules for this seems overkill. But we do want the ability to dump and load cubes.
  • What about grdinfo for 3-D cubes? We currently have no way to determine the equivalent info for cubes that we create. I see -Q is available there too.

@PaulWessel PaulWessel added the new feature PR that implements a new feature or capability in GMT label Dec 16, 2020
@PaulWessel PaulWessel self-assigned this Dec 16, 2020
@PaulWessel
Copy link
Member Author

Alternatively, no -Q is needed for grdinfo and grd2xyz since we could presumably check if a given netCDF file is a grid or cube, and then take action. As for xyz2grd one has to specify -Rw/e/s/n -Idx/dy, but if a cube this needs to be -Rw/e/s/n/z0/z1 -Idx/dxy/dz so perhaps there is no need for -Q there either. What do you think, @joa-quim ?

@joa-quim
Copy link
Member

grd2xyz and xyz2grd so they can handle 4-D data

What is the interest of this? Why care to convert a 4-D to xyzt and back?
A more useful tool would a stacker, which would read a pack of grids and stack them into a 3-D nc file.

@PaulWessel
Copy link
Member Author

grdinterpolate grid1 grid2 ... -Zlevels -Gcube.nc will do the stacking (in next commit).
We cared enough to convert grd <--> xyz so I guess it is not that different to do cube <--> xyzw ?

@joa-quim
Copy link
Member

joa-quim commented Dec 16, 2020

We cared enough to convert grd <--> xyz

Yes, > 20 years ago when people still wrote grids manually. I fail to see why spending time coding a 4D analog

@PaulWessel
Copy link
Member Author

OK, we can wait to see if there is any demand. What about grdinfo for cubes?

src/gmt_nc.c Outdated

/* Depending on row_order, write y-coordinate array bottom-to-top or top-to-bottom */
if (HH->row_order == k_nc_start_south) {
for (row = 0; row < header->n_rows; row++) xy[row] = (double) gmt_M_col_to_x (GMT, row, header->wesn[YLO], header->wesn[YHI], header->inc[GMT_Y], 0.5 * header->registration, header->n_rows);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please break line. Lines like these are an horror to debug.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

OK, done in my workspace - not ready to commit yet.

@joa-quim
Copy link
Member

grdinfo for cubes seems useful.

This does not work since registration is a problem.  Now done in grdinterpolate instead.
@PaulWessel
Copy link
Member Author

PaulWessel commented Dec 19, 2020

Pinging @remkos to have a look if he has time. In addition to the email I sent, also see the issue with lack of assigning variable names and units a few posts above.

@remkos
Copy link
Contributor

remkos commented Dec 26, 2020

I was looking at your proposal for units. As far as I remember (but I may have done that in other programs), if one adds "[m]" to the x-name then "m" is used as the unit. This should also hold for y-name, z-name and any other specified units, of course.

But yes, this would be rather "hidden" way of doing things and probably a more direct way of specifying the units, as you suggested above would be better. Or maybe support both.

Indeed, there was no way to specify the variable name. It was always x and y, unless the projection was geographical and lon and lat where used. This is not required by COARDS, but I have seen applications react to it appropriately, even without the units like degrees_east, degrees_north or the appropriate standard_name. Still, what is required by COARDS is that the dimensions and the coordinate axes are called the same. So the dimensions and the coordinate arrays are called lon and lat, consistently.

Now checking the code:

  • Yes, gmtnc_put_units does split off the [units] part from the name_units variable which can be either x_units or y_units. So if in the present form you would specify "x-unit" as "temperature [K]" it would write long_name = "temperature" and units = "K".
  • There are also some provisions for time axes. But it does not look like that I made any provisions to put the time references in correctly, e.g. "seconds since 2000-01-01 00:00:00".

@PaulWessel
Copy link
Member Author

It is great that name [unit] is parsed like that - I guess I thought it was a special thing for [degrees_longitude] etc. and not general. So we can improve the docs on that for sure.

In the process, change greenspline -D to -Z but backwards compatible so that it and grdinterpolate can take -Dinformation to set the various grid/cube variables.  Updated scripts using -D and ensure the "name [unit]" parsing works for x,y,z, and d.  Renamed +zsunit to +ddataunit for both grids and cube but this is backwards compatible for grids of course.
@PaulWessel
Copy link
Member Author

Since (re-)discovering the ability to specify the name of a dimension or data and add the unit in brackets, the proposal above has been simplified, and implemented. We now have, via the -Dinfo setting, these new things:

  1. +ddname sets the name and unit of the data value (formerly just +zname for grids - this is still backwards compatible) [z]
  2. +zname sets the 3rd dimension name and unit for cubes [z or time]
  3. +vvarname names the netCDF data variable [z or cube]

With that I think we are good. I updated gspline_4.sh to use -D (which I added to greenspline) via the command

gmt greenspline -R12/32/0/6/5/10 -I0.25 -G3D_cube.grdTable_5_23.txt -Sr0.85 -Z5 -D+x"x-distance [km]"+y"y-distance [km]"+z"z-distance [km]"+d"Uranium Oxide [%]"+vuoxide"

and now ncdump -h gives

variables:
	double x(x) ;
		x:long_name = "x-distance" ;
		x:units = "km" ;
		x:actual_range = 12., 32. ;
	double y(y) ;
		y:long_name = "y-distance" ;
		y:units = "km" ;
		y:actual_range = 0., 6. ;
	double z(z) ;
		z:long_name = "z-distance" ;
		z:units = "km" ;
		z:actual_range = 5., 10. ;
	float uoxide(z, y, x) ;
		uoxide:long_name = "Uranium Oxide" ;
		uoxide:units = "%" ;
		uoxide:_FillValue = NaNf ;
		uoxide:actual_range = 0., 23.2000007629395 ;

So, the -Dinfo was added to both greenspline and grdinterpolate. Since greenspline already used -D I moved that option to -Z but we handle this with backwards compatibility. All tests work fine - I think this PR is now good to be approved and merged.

Copy link
Member

@joa-quim joa-quim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Tests pass for me too.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature PR that implements a new feature or capability in GMT
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants