Possible workaround for HDF5 growing memory issue

[pzelasko]

Intended as a workaround for growing memory issues when using HDF5 with larger datasets.

[csukuangfj]

Shall we reset self.batch_counter in case it overflows if it runs for enough time?

[danpovey]

Python uses special integers that can never overflow, I believe.

[pzelasko]

I cleaned up the code and added some docs. I'd appreciate if somebody can test it out on a large scale training and confirm that it does help; I was only able to test on small-to-medium sized problems so far.

[pzelasko]

In case anybody reads this: this workaround doesn't completely solve the issue, and the memory keeps growing, just slower. There also seem to be random spikes in memory usage around the time the HDF5 files are open/closed.

I generally don't advise using LilcomHdf5Writer for data larger than ~1k hours as it will likely blow up the memory at some point.

[desh2608]

@pzelasko which is the recommended storage type for large data (5-10k hours)?

[pzelasko]

The default one (LilcomChunkyWriter) should work as well as HDF5 but without memory leaks.

[cyrta]

And what about cloud storage ?

[pzelasko]

The recent PRs with WebDataset integration work really well with that; please refer to #582, #599, and #602. You'll need to read the documentation in the code because we don't have a high-level tutorial for these workflows yet.

[pzelasko]

... alternatively, there is also LilcomUrlWriter for features and AudioSource(type="url", ...) for audio which you can use with any cloud storage supported by smart_open library. It will be much less I/O efficient because it will spawn a new connection for each item, but for some types of use-cases might be viable. But WebDataset-based flow would be roughly 50-100x faster.