Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

HDF5 interface: add write_dict_to_hdf() and read_dict_from_hdf() #1247

Merged
merged 31 commits into from
Dec 20, 2023

Conversation

jan-janssen
Copy link
Member

@jan-janssen jan-janssen commented Dec 16, 2023

This pull request refactors the helper functions in the pyiron_base.storage module and adds the write_dict_to_hdf() and read_dict_from_hdf() functions. The write_dict_to_hdf() function takes a dictionary where the keys represent the path in the HDF5 file and the values represent the objects which should be stored in these nodes. The corresponding HDF5 groups are created automatically. The read_dict_from_hdf() function reads the nodes from a given HDF5 group and subgroups can be included using the optional group_paths parameter. The function returns a hierarchical dictionary which can be directly converted to a DataContainer.

@jan-janssen jan-janssen added the format_black reformat the code using the black standard label Dec 16, 2023
@jan-janssen jan-janssen changed the title HDF5 move json conversion check to separate function HDF5 interface: Introduce write_dict_to_hdf() to write a whole dictionary to an HDF5 file at once Dec 18, 2023
@jan-janssen
Copy link
Member Author

@pmrv As discussed in the pyiron meeting, I plan to go ahead and merge this so we can focus on defining the interface in a separate pull request.

pyiron_base/storage/helper_functions.py Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
@jan-janssen
Copy link
Member Author

@pmrv I fixed the remarks and would like to merge this. So if you have time to take another look that would be great.

@jan-janssen jan-janssen added format_black reformat the code using the black standard and removed format_black reformat the code using the black standard labels Dec 20, 2023
@jan-janssen jan-janssen changed the title HDF5 interface: Introduce write_dict_to_hdf() to write a whole dictionary to an HDF5 file at once HDF5 interface: add write_dict_to_hdf() and read_dict_from_hdf() Dec 20, 2023
@jan-janssen
Copy link
Member Author

While I made a couple more changes to add the read_dict_from_hdf() function - still the pull request is now again ready for review.

Copy link
Contributor

@pmrv pmrv left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I left a number of small remarks, but can be merged after those are considered without re-review.

The only bigger thing I've seen is the slash parameter. While it was present before this PR introduces a lot of code now that offers it, but afaik we never use this, so it can be just removed completely imo

pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
pyiron_base/storage/helper_functions.py Outdated Show resolved Hide resolved
jan-janssen and others added 11 commits December 20, 2023 16:21
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
Co-authored-by: Marvin Poul <poul@mpie.de>
# Conflicts:
#	pyiron_base/storage/helper_functions.py
@jan-janssen
Copy link
Member Author

The only bigger thing I've seen is the slash parameter. While it was present before this PR introduces a lot of code now that offers it, but afaik we never use this, so it can be just removed completely imo

I would like to make the slash parameter as well as the compression setting accessible to the outside once we release the stand-alone HDF5 interface, so I already added it to the central functions in the helper class.

@jan-janssen jan-janssen merged commit 2964d4d into main Dec 20, 2023
23 of 24 checks passed
@delete-merged-branch delete-merged-branch bot deleted the h5json branch December 20, 2023 17:26
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
format_black reformat the code using the black standard
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants