Refactored the code, used datasets to load data, decreased padding from 30% to 5%, unified language model training logic into trainer to reduce redundant code, and supported a more reasonable config format.
Refactored the code, used datasets to load data, decreased padding from 30% to 5%, unified language model training logic into trainer to reduce redundant code, and supported a more reasonable config format.