Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

add an option to merge data to one H5 file #1119

Merged
merged 1 commit into from
Jan 19, 2023
Merged

add an option to merge data to one H5 file #1119

merged 1 commit into from
Jan 19, 2023

Conversation

njzjz
Copy link
Member

@njzjz njzjz commented Jan 17, 2023 •
edited
Loading

Fix #617.

@njzjz
add an option to merge data to one H5 file
e870cda 
Signed-off-by: Jinzhe Zeng <jinzhe.zeng@rutgers.edu>
@njzjz njzjz linked an issue Jan 17, 2023 that may be closed by this pull request
[Feature Request] Use HDF5 instead of numpy files #617
Closed
@njzjz njzjz marked this pull request as ready for review January 17, 2023 21:16
@njzjz njzjz mentioned this pull request Jan 17, 2023
Closed
@njzjz
Copy link
Member Author

njzjz commented Jan 18, 2023

Below is the test on my systems:

INFO:dpgen:Combining 45929 training systems to iter.000007/00.train/data.hdf5...
INFO:dpgen:-------------------------iter.000007 task 01--------------------------
2023-01-17 18:01:36,925 - INFO : info:check_all_finished: False
2023-01-17 18:01:36,925 - INFO : remote path: /scratch/jz748/dpgen_workdir/5351930e3552f5d9c883518f7e9e839f81cce2aa
2023-01-17 18:04:23,468 - INFO : job: 82764aa11f3b7d0d57c49a263df5405f75da2720 submit; job_id is 24121180

A 1.7GB file is generated:

-rw-r--r--  1 jz748 york 1.7G Jan 17 17:59 data.hdf5

@wanghan-iapcm wanghan-iapcm merged commit fb8668b into deepmodeling:devel Jan 19, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Vibsteamer Vibsteamer Vibsteamer approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature Request] Use HDF5 instead of numpy files
3 participants
@njzjz @Vibsteamer @wanghan-iapcm