This repository is the winning solution to 2021 CCF Big Data & Computing Intelligence Contest (2021 CCF BDCI) - "Large BERT Training Challange Cup".
Team: PKU-DAIR. Members: Xupeng Miao, Xiaonan Nie, Yujie Wang.
Our Hetu team members have won both 1st and 3rd in the contest. And this solution is based on PyTorch and achieves the hidden size of 2080. We also provide an implementation over Hetu distributed deep learning system, which achieves an amazing hidden size of 2128!
- python=3.6.13
./train.sh
