Allow training without loading full dataset into memory

[shiyu1994]

Summary

Currently LightGBM loads the full dataset into memory for training. Allow training without loading the (full) dataset into memory should be very useful.

Motivation

As datasets grow larger, training without loading the full dataset into memory is very necessary. See e.g., #5055

Description

References

An experimental feature from XGBoost, https://xgboost.readthedocs.io/en/stable/tutorials/external_memory.html

[shiyu1994]

Closed in favor of being in #2302. We decided to keep all feature requests in one place.

Welcome to contribute this feature! Please re-open this issue (or post a comment if you are not a topic starter) if you are actively working on implementing this feature.

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed.
To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues
including a reference to this.