Tools long double updates #522

byrnHDF · 2021-03-26T12:33:31Z

Added tests for tools and java testing.

…evelop

lrknox · 2021-04-06T03:11:38Z

This is the output on jelly and hedgehog from h5dump of tldouble.h5 with current source from Allen's fork:

Jelly (little endian) output
[lrknox@jelly tools]$ src/h5dump/.libs/h5dump ../../tools/testfiles/tldouble.h5
HDF5 "../../tools/testfiles/tldouble.h5" {
GROUP "/" {
DATASET "dset" {
DATATYPE H5T_NATIVE_LDOUBLE
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
DATA {
(0): 1, 2, 3
}
...

hedgehog (big endian) output
-bash-5.0$ src/h5dump/.libs/h5dump ../../tools/testfiles/tldouble.h5
HDF5 "../../tools/testfiles/tldouble.h5" {
GROUP "/" {
DATASET "dset" {
DATATYPE 128-bit little-endian floating-point
DATASPACE SIMPLE { ( 3 ) / ( 3 ) }
DATA {
(0): 1, 2, 3
}
...

epourmal · 2021-04-06T11:35:00Z

Hmmm....output should be the same. This doesn’t sound right.

…

Sent from my iPhone On Apr 5, 2021, at 10:11 PM, Larry Knox ***@***.***> wrote: This is the output on jelly and hedgehog from h5dump of tldouble.h5 with current source from Allen's fork: Jelly (little endian) output ***@***.*** tools]$ src/h5dump/.libs/h5dump ../../tools/testfiles/tldouble.h5 HDF5 "../../tools/testfiles/tldouble.h5" { GROUP "/" { DATASET "dset" { DATATYPE H5T_NATIVE_LDOUBLE DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 1, 2, 3 } ... hedgehog (big endian) output -bash-5.0$ src/h5dump/.libs/h5dump ../../tools/testfiles/tldouble.h5 HDF5 "../../tools/testfiles/tldouble.h5" { GROUP "/" { DATASET "dset" { DATATYPE 128-bit little-endian floating-point DATASPACE SIMPLE { ( 3 ) / ( 3 ) } DATA { (0): 1, 2, 3 } ... — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#522 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADLFT3KMKCZCT7MLDCLM7S3THJ3XVANCNFSM4Z3KACHQ>.

byrnHDF · 2021-04-06T11:40:14Z

Yes the output is correct and the test should pass.
This was the conclusion of the developers on Friday, that the tools should print NATIVE if the datatype matches that platform's NATIVE type, else print what the library knows about the type.
I might need to add more characteristics like precision.

epourmal · 2021-04-06T12:12:11Z

This decision is wrong. To everyone: native type reflects language type in memory on a particular platform. Information in the file is self-descriptive and should be portable. Native is not self-descriptive and portable. Tools should reflect that what is in the file is not supported.

…

Sent from my iPhone On Apr 6, 2021, at 6:40 AM, Allen Byrne ***@***.***> wrote: Yes the output is correct and the test should pass. This was the conclusion of the developers on Friday, that the tools should print NATIVE if the datatype matches that platform's NATIVE type, else print what the library knows about the type. I might need to add more characteristics like precision. — You are receiving this because you commented. Reply to this email directly, view it on GitHub<#522 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/ADLFT3M6KQMW2XMWSONMKETTHLXKZANCNFSM4Z3KACHQ>.

byrnHDF · 2021-04-07T14:03:30Z

I will revert back to the removal of NATIVE_LDOUBLE in h5dump and h5ls. I will create an issue for the other native types in tools.
I will also add a precision value to the output of integers and floats.

byrnHDF · 2021-04-08T12:58:29Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

lrknox · 2021-04-13T02:19:33Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

byrnHDF · 2021-04-13T12:00:04Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

Also note that the n-bit h5dump test now shows that the datatype is of 3-bit precision:
DATATYPE 32-bit little-endian integer 3-bit precision

qkoziol · 2021-04-13T15:46:52Z

tools/lib/h5tools_dump.c

-                /* print size, order, and sign  */
-                h5tools_str_append(buffer, "%lu-bit%s%s integer", (unsigned long)(8 * H5Tget_size(type)),
-                                   order_s, sign_s);
+                /* print size, order, sign, and precision */


This is helpful, and probably sufficient for integers.

qkoziol · 2021-04-13T15:47:54Z

tools/lib/h5tools_dump.c

-                /* print size and byte order */
-                h5tools_str_append(buffer, "%lu-bit%s floating-point", (unsigned long)(8 * H5Tget_size(type)),
-                                   order_s);
+                /* print size. byte order, and precision */


This is helpful, and not sufficient for floating-point types. There's a lot more details for floating point types: mantissa, sign bit, etc.

Yes, but how much detail is needed? h5ls does print more and I could make h5dump match h5ls - at the very least.

What's the output from h5ls look like?

Opened "tldouble.h5" with sec2 driver.
dset Dataset {3/3}
Location: 1:800
Links: 1
Storage: 48 logical bytes, 48 allocated bytes, 100.00% utilization
Type: 128-bit little-endian floating-point
(80 bits of precision beginning at bit 0)
(48 zero bits at bit 80)
(significant for 64 bits at bit 0, no normalization)
(exponent for 15 bits at bit 64, bias is 0x3fff)
(sign bit at 79)

qkoziol · 2021-04-13T15:50:20Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

Some of this is in the right direction (more information for custom atomic types), but the H5T_NATIVE_LDOUBLE issue still isn't resolved. And now there's doxygen and Perl changes that should be in another PR, not here.

byrnHDF · 2021-04-13T16:06:37Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

Some of this is in the right direction (more information for custom atomic types), but the H5T_NATIVE_LDOUBLE issue still isn't resolved.

According to Elena, NATIVE is a memory type. She has stated that h5dump/h5ls should display the file type. That reasoning suggests that h5dump/h5ls should eliminate the display of the NATIVE types and always show the file type. I agree since the NATIVE type is dependent on platform/compiler.

And now there's doxygen and Perl changes that should be in another PR, not here.

Well this PR is taking so long, I accidentally committed files!

qkoziol · 2021-04-13T16:27:00Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

Some of this is in the right direction (more information for custom atomic types), but the H5T_NATIVE_LDOUBLE issue still isn't resolved.

According to Elena, NATIVE is a memory type. She has stated that h5dump/h5ls should display the file type. That reasoning suggests that h5dump/h5ls should eliminate the display of the NATIVE types and always show the file type. I agree since the NATIVE type is dependent on platform/compiler.

That suggests that all the "native" datatype output should be removed. Is that true?

qkoziol · 2021-04-13T16:27:29Z

And now there's doxygen and Perl changes that should be in another PR, not here.

Well this PR is taking so long, I accidentally committed files!

OK, please back out the accidental commit then.

epourmal · 2021-04-13T17:19:30Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

Some of this is in the right direction (more information for custom atomic types), but the H5T_NATIVE_LDOUBLE issue still isn't resolved.

According to Elena, NATIVE is a memory type. She has stated that h5dump/h5ls should display the file type. That reasoning suggests that h5dump/h5ls should eliminate the display of the NATIVE types and always show the file type. I agree since the NATIVE type is dependent on platform/compiler.

That suggests that all the "native" datatype output should be removed. Is that true?

Please remove NATIVE type from tools' output and merge the fix.

HDF5 tools display properties of data as stored in the file. If someone thinks that "NATIVE" can be used to describe portable data in the file, please justify, how "NATIVE" is portable. Thank you!

qkoziol · 2021-04-13T17:30:07Z

The long double h5dump output now appears as:
DATATYPE 128-bit little-endian floating-point 80-bit precision

@koziol does this change satisfy your request?

Some of this is in the right direction (more information for custom atomic types), but the H5T_NATIVE_LDOUBLE issue still isn't resolved.

According to Elena, NATIVE is a memory type. She has stated that h5dump/h5ls should display the file type. That reasoning suggests that h5dump/h5ls should eliminate the display of the NATIVE types and always show the file type. I agree since the NATIVE type is dependent on platform/compiler.

That suggests that all the "native" datatype output should be removed. Is that true?

Please remove NATIVE type from tools' output and merge the fix.

HDF5 tools display properties of data as stored in the file. If someone thinks that "NATIVE" can be used to describe portable data in the file, please justify, how "NATIVE" is portable. Thank you!

I think that's good reasoning. :-) (Although a lot more work on this PR before it's merged)

byrnHDF · 2021-04-13T19:07:38Z

I agree, remove NATIVE display output.
Removed non long double concerned files.
Q: Should this be merged and then followup with a PR to remove the other NATIVE display output or combine both?

byrnHDF · 2021-04-14T13:01:37Z

Created issue #551 for fixing tools to only display file datatypes.
Also created HDFFV-11236 for the h5dump BNF specification.

byrnHDF added 30 commits January 7, 2021 07:15

OESS-98 convert plugin option to FetchContent, add tests

23e0df3

Fixes for pkcfg files because of plugin option

84293f5

OESS-98 fix tools test for plugins

544bd61

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

ffb182f

…evelop

Keep doxygen comments under 100 chars long - format hint

2561b6b

Whitespace

1c73615

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

b71a853

…evelop

HDFFV-11144 - Reclassify CMake messages

b68973c

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

f64497d

…evelop

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

c854ffe

…evelop

HDFFV-11099/11100 added help text

0f90d26

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

20757d6

…evelop

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

d161733

…evelop

Reworked switch statement to compare string instead

f03b095

Fix typo

bd52065

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

238c060

…evelop

Update CDash mode

f12b2ae

Correct name of threadsafe

673aac8

Correct option name

97df6db

Undo accidental commit

e48f758

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

2f364d8

…evelop

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

f42f02b

…evelop

Note LLVM 10 to 11 format default changes

5b48b33

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

9927372

…evelop

Update format plugin

ad61d57

Undo clang-format version 11 changes

613bc7a

One more correction

0114098

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

a685efa

…evelop

Merge branch 'develop' of https://github.com/HDFGroup/hdf5.git into d…

78f25ac

…evelop

Update supported platforms

1899ba6

lrknox approved these changes Apr 6, 2021

View reviewed changes

byrnHDF and others added 4 commits April 7, 2021 09:04

HDFFV-11229 revert back to removal of NATIVE_LDOUBLE in tools output

2385150

Committing clang-format changes

c17ca54

Update release note

f05c84e

Update attribute type of ref file

1e6224c

byrnHDF added 3 commits April 12, 2021 13:49

Change source of ninja for macs

baf0fd3

try port instead of brew

6d780f6

Recommended is to use brew.

c2cf490

qkoziol reviewed Apr 13, 2021

View reviewed changes

epourmal approved these changes Apr 13, 2021

View reviewed changes

byrnHDF added 2 commits April 13, 2021 14:02

Undo non long double changes

2e0f725

remove unneeded file

e97e2e8

lrknox merged commit e21f7aa into HDFGroup:develop Apr 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Tools long double updates #522

Tools long double updates #522

byrnHDF commented Mar 26, 2021

lrknox commented Apr 6, 2021

epourmal commented Apr 6, 2021 via email

byrnHDF commented Apr 6, 2021

epourmal commented Apr 6, 2021 via email

byrnHDF commented Apr 7, 2021

byrnHDF commented Apr 8, 2021

lrknox commented Apr 13, 2021

byrnHDF commented Apr 13, 2021

qkoziol Apr 13, 2021

qkoziol Apr 13, 2021

byrnHDF Apr 13, 2021

qkoziol Apr 13, 2021

byrnHDF Apr 13, 2021

qkoziol commented Apr 13, 2021

byrnHDF commented Apr 13, 2021 •

edited

Loading

qkoziol commented Apr 13, 2021

qkoziol commented Apr 13, 2021

epourmal commented Apr 13, 2021

qkoziol commented Apr 13, 2021

byrnHDF commented Apr 13, 2021

byrnHDF commented Apr 14, 2021

Tools long double updates #522

Tools long double updates #522

Conversation

byrnHDF commented Mar 26, 2021

lrknox commented Apr 6, 2021

epourmal commented Apr 6, 2021 via email

byrnHDF commented Apr 6, 2021

epourmal commented Apr 6, 2021 via email

byrnHDF commented Apr 7, 2021

byrnHDF commented Apr 8, 2021

lrknox commented Apr 13, 2021

byrnHDF commented Apr 13, 2021

qkoziol Apr 13, 2021

Choose a reason for hiding this comment

qkoziol Apr 13, 2021

Choose a reason for hiding this comment

byrnHDF Apr 13, 2021

Choose a reason for hiding this comment

qkoziol Apr 13, 2021

Choose a reason for hiding this comment

byrnHDF Apr 13, 2021

Choose a reason for hiding this comment

qkoziol commented Apr 13, 2021

byrnHDF commented Apr 13, 2021 • edited Loading

qkoziol commented Apr 13, 2021

qkoziol commented Apr 13, 2021

epourmal commented Apr 13, 2021

qkoziol commented Apr 13, 2021

byrnHDF commented Apr 13, 2021

byrnHDF commented Apr 14, 2021

byrnHDF commented Apr 13, 2021 •

edited

Loading