Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Recover when GPU OOMs #48

Closed
pseudotensor opened this issue Apr 19, 2023 · 2 comments
Closed

Recover when GPU OOMs #48

pseudotensor opened this issue Apr 19, 2023 · 2 comments

Comments

@pseudotensor
Copy link
Collaborator 

torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 22.20 GiB total capacity; 20.67 GiB already allocated; 4.12 MiB free; 21.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

brings app down, no longer can generate. Protect against GPU OOM or at least recover without hanging.
@pseudotensor
Copy link
Collaborator Author 


Downloading (…)l-00004-of-00005.bin:  52%|█████▏    | 2.63G/5.02G [00:43<00:40, 59.4MB/s]

Downloading (…)l-00004-of-00005.bin:  53%|█████▎    | 2.64G/5.02G [00:43<00:49, 47.9MB/s]

Downloading (…)l-00004-of-00005.bin:  53%|█████▎    | 2.66G/5.02G [00:44<00:47, 50.0MB/s]

Downloading (…)l-00004-of-00005.bin:  53%|█████▎    | 2.67G/5.02G [00:44<00:44, 52.6MB/s]

Downloading (…)l-00004-of-00005.bin:  54%|█████▎    | 2.69G/5.02G [00:44<00:37, 62.3MB/s]

Downloading (…)l-00004-of-00005.bin:  54%|█████▍    | 2.71G/5.02G [00:44<00:38, 60.4MB/s]

Downloading (…)l-00004-of-00005.bin:  54%|█████▍    | 2.73G/5.02G [00:44<00:29, 78.1MB/s]

Downloading (…)l-00004-of-00005.bin:  55%|█████▍    | 2.74G/5.02G [00:45<00:34, 66.5MB/s]

Downloading (…)l-00004-of-00005.bin:  55%|█████▍    | 2.76G/5.02G [00:45<00:34, 65.3MB/s]

Downloading (…)l-00004-of-00005.bin:  55%|█████▌    | 2.77G/5.02G [00:45<00:34, 65.3MB/s]

Downloading (…)l-00004-of-00005.bin:  56%|█████▌    | 2.79G/5.02G [00:45<00:34, 65.5MB/s]

Downloading (…)l-00004-of-00005.bin:  56%|█████▌    | 2.80G/5.02G [00:46<00:34, 64.5MB/s]

Downloading (…)l-00004-of-00005.bin:  56%|█████▌    | 2.81G/5.02G [00:46<00:38, 57.3MB/s]

Downloading (…)l-00004-of-00005.bin:  56%|█████▌    | 2.82G/5.02G [00:46<00:37, 59.0MB/s]

Downloading (…)l-00004-of-00005.bin:  57%|█████▋    | 2.84G/5.02G [00:46<00:35, 61.9MB/s]

Downloading (…)l-00004-of-00005.bin:  57%|█████▋    | 2.85G/5.02G [00:47<00:36, 59.8MB/s]

Downloading (…)l-00004-of-00005.bin:  57%|█████▋    | 2.87G/5.02G [00:47<00:31, 68.7MB/s]

Downloading (…)l-00004-of-00005.bin:  57%|█████▋    | 2.88G/5.02G [00:47<00:37, 57.4MB/s]

Downloading (…)l-00004-of-00005.bin:  58%|█████▊    | 2.90G/5.02G [00:47<00:33, 63.2MB/s]

Downloading (…)l-00004-of-00005.bin:  58%|█████▊    | 2.92G/5.02G [00:48<00:35, 59.0MB/s]

Downloading (…)l-00004-of-00005.bin:  59%|█████▊    | 2.94G/5.02G [00:48<00:29, 70.7MB/s]

Downloading (…)l-00004-of-00005.bin:  59%|█████▊    | 2.95G/5.02G [00:48<00:29, 70.2MB/s]

Downloading (…)l-00004-of-00005.bin:  59%|█████▉    | 2.97G/5.02G [00:48<00:32, 62.4MB/s]

Downloading (…)l-00004-of-00005.bin:  59%|█████▉    | 2.98G/5.02G [00:49<00:46, 43.7MB/s]

Downloading (…)l-00004-of-00005.bin:  60%|█████▉    | 2.99G/5.02G [00:49<00:40, 50.4MB/s]

Downloading (…)l-00004-of-00005.bin:  60%|█████▉    | 3.00G/5.02G [00:49<00:39, 51.2MB/s]

Downloading (…)l-00004-of-00005.bin:  60%|██████    | 3.01G/5.02G [00:49<00:36, 55.5MB/s]

Downloading (…)l-00004-of-00005.bin:  60%|██████    | 3.03G/5.02G [00:50<00:32, 61.1MB/s]

Downloading (…)l-00004-of-00005.bin:  61%|██████    | 3.04G/5.02G [00:50<00:33, 58.9MB/s]

Downloading (…)l-00004-of-00005.bin:  61%|██████    | 3.06G/5.02G [00:50<00:25, 77.6MB/s]

Downloading (…)l-00004-of-00005.bin:  61%|██████▏   | 3.07G/5.02G [00:50<00:25, 76.5MB/s]

Downloading (…)l-00004-of-00005.bin:  62%|██████▏   | 3.09G/5.02G [00:50<00:23, 82.3MB/s]

Downloading (…)l-00004-of-00005.bin:  62%|██████▏   | 3.10G/5.02G [00:51<00:30, 62.6MB/s]

Downloading (…)l-00004-of-00005.bin:  62%|██████▏   | 3.11G/5.02G [00:51<00:28, 65.7MB/s]

Downloading (…)l-00004-of-00005.bin:  62%|██████▏   | 3.12G/5.02G [00:51<00:29, 64.0MB/s]

Downloading (…)l-00004-of-00005.bin:  63%|██████▎   | 3.15G/5.02G [00:51<00:24, 77.7MB/s]

Downloading (…)l-00004-of-00005.bin:  63%|██████▎   | 3.16G/5.02G [00:51<00:29, 62.9MB/s]

Downloading (…)l-00004-of-00005.bin:  63%|██████▎   | 3.17G/5.02G [00:52<00:27, 66.9MB/s]

Downloading (…)l-00004-of-00005.bin:  63%|██████▎   | 3.18G/5.02G [00:52<00:26, 68.6MB/s]

Downloading (…)l-00004-of-00005.bin:  64%|██████▎   | 3.19G/5.02G [00:52<00:30, 59.0MB/s]

Downloading (…)l-00004-of-00005.bin:  64%|██████▍   | 3.21G/5.02G [00:52<00:25, 72.1MB/s]

Downloading (…)l-00004-of-00005.bin:  64%|██████▍   | 3.22G/5.02G [00:52<00:30, 59.8MB/s]

Downloading (…)l-00004-of-00005.bin:  64%|██████▍   | 3.23G/5.02G [00:53<00:33, 53.9MB/s]

Downloading (…)l-00004-of-00005.bin:  65%|██████▍   | 3.24G/5.02G [00:53<00:36, 48.1MB/s]

Downloading (…)l-00004-of-00005.bin:  65%|██████▍   | 3.25G/5.02G [00:53<00:38, 46.3MB/s]

Downloading (…)l-00004-of-00005.bin:  65%|██████▌   | 3.27G/5.02G [00:53<00:30, 56.7MB/s]

Downloading (…)l-00004-of-00005.bin:  65%|██████▌   | 3.28G/5.02G [00:54<00:30, 56.7MB/s]

Downloading (…)l-00004-of-00005.bin:  66%|██████▌   | 3.30G/5.02G [00:54<00:23, 73.2MB/s]

Downloading (…)l-00004-of-00005.bin:  66%|██████▌   | 3.31G/5.02G [00:54<00:23, 73.8MB/s]

Downloading (…)l-00004-of-00005.bin:  66%|██████▋   | 3.33G/5.02G [00:54<00:19, 85.8MB/s]

Downloading (…)l-00004-of-00005.bin:  67%|██████▋   | 3.34G/5.02G [00:54<00:23, 70.3MB/s]

Downloading (…)l-00004-of-00005.bin:  67%|██████▋   | 3.37G/5.02G [00:55<00:20, 79.2MB/s]

Downloading (…)l-00004-of-00005.bin:  67%|██████▋   | 3.38G/5.02G [00:55<00:24, 67.5MB/s]

Downloading (…)l-00004-of-00005.bin:  68%|██████▊   | 3.40G/5.02G [00:55<00:20, 78.2MB/s]

Downloading (…)l-00004-of-00005.bin:  68%|██████▊   | 3.42G/5.02G [00:55<00:19, 83.1MB/s]

Downloading (…)l-00004-of-00005.bin:  68%|██████▊   | 3.43G/5.02G [00:55<00:23, 68.6MB/s]

Downloading (…)l-00004-of-00005.bin:  69%|██████▉   | 3.45G/5.02G [00:56<00:23, 67.8MB/s]

Downloading (…)l-00004-of-00005.bin:  69%|██████▉   | 3.46G/5.02G [00:56<00:28, 55.5MB/s]

Downloading (…)l-00004-of-00005.bin:  69%|██████▉   | 3.48G/5.02G [00:56<00:23, 64.8MB/s]

Downloading (…)l-00004-of-00005.bin:  70%|██████▉   | 3.49G/5.02G [00:57<00:32, 46.8MB/s]

Downloading (…)l-00004-of-00005.bin:  70%|███████   | 3.51G/5.02G [00:57<00:26, 56.3MB/s]

Downloading (…)l-00004-of-00005.bin:  70%|███████   | 3.52G/5.02G [00:57<00:28, 52.4MB/s]

Downloading (…)l-00004-of-00005.bin:  71%|███████   | 3.54G/5.02G [00:58<00:26, 55.7MB/s]

Downloading (…)l-00004-of-00005.bin:  71%|███████   | 3.55G/5.02G [00:58<00:27, 53.5MB/s]

Downloading (…)l-00004-of-00005.bin:  71%|███████▏  | 3.58G/5.02G [00:58<00:23, 61.0MB/s]

Downloading (…)l-00004-of-00005.bin:  72%|███████▏  | 3.59G/5.02G [00:58<00:27, 52.5MB/s]

Downloading (…)l-00004-of-00005.bin:  72%|███████▏  | 3.61G/5.02G [00:59<00:20, 69.1MB/s]

Downloading (…)l-00004-of-00005.bin:  72%|███████▏  | 3.62G/5.02G [00:59<00:22, 63.4MB/s]

Downloading (…)l-00004-of-00005.bin:  73%|███████▎  | 3.64G/5.02G [00:59<00:17, 79.2MB/s]

Downloading (…)l-00004-of-00005.bin:  73%|███████▎  | 3.65G/5.02G [00:59<00:18, 74.3MB/s]

Downloading (…)l-00004-of-00005.bin:  73%|███████▎  | 3.67G/5.02G [00:59<00:17, 74.8MB/s]

Downloading (…)l-00004-of-00005.bin:  73%|███████▎  | 3.68G/5.02G [01:00<00:20, 65.8MB/s]

Downloading (…)l-00004-of-00005.bin:  74%|███████▍  | 3.70G/5.02G [01:00<00:19, 68.3MB/s]

Downloading (…)l-00004-of-00005.bin:  74%|███████▍  | 3.72G/5.02G [01:00<00:17, 75.9MB/s]

Downloading (…)l-00004-of-00005.bin:  74%|███████▍  | 3.73G/5.02G [01:00<00:21, 58.7MB/s]

Downloading (…)l-00004-of-00005.bin:  75%|███████▍  | 3.75G/5.02G [01:01<00:20, 62.0MB/s]

Downloading (…)l-00004-of-00005.bin:  75%|███████▌  | 3.76G/5.02G [01:01<00:21, 56.9MB/s]

Downloading (…)l-00004-of-00005.bin:  75%|███████▌  | 3.79G/5.02G [01:01<00:16, 74.3MB/s]

Downloading (…)l-00004-of-00005.bin:  76%|███████▌  | 3.80G/5.02G [01:01<00:17, 70.9MB/s]

Downloading (…)l-00004-of-00005.bin:  76%|███████▌  | 3.81G/5.02G [01:01<00:15, 76.6MB/s]

Downloading (…)l-00004-of-00005.bin:  76%|███████▌  | 3.82G/5.02G [01:02<00:16, 73.5MB/s]

Downloading (…)l-00004-of-00005.bin:  76%|███████▋  | 3.83G/5.02G [01:02<00:17, 66.5MB/s]

Downloading (…)l-00004-of-00005.bin:  77%|███████▋  | 3.84G/5.02G [01:02<00:20, 57.8MB/s]

Downloading (…)l-00004-of-00005.bin:  77%|███████▋  | 3.85G/5.02G [01:02<00:22, 52.3MB/s]

Downloading (…)l-00004-of-00005.bin:  77%|███████▋  | 3.86G/5.02G [01:03<00:40, 28.7MB/s]

Downloading (…)l-00004-of-00005.bin:  77%|███████▋  | 3.88G/5.02G [01:03<00:29, 38.8MB/s]

Downloading (…)l-00004-of-00005.bin:  78%|███████▊  | 3.89G/5.02G [01:04<00:33, 33.7MB/s]

Downloading (…)l-00004-of-00005.bin:  78%|███████▊  | 3.91G/5.02G [01:04<00:25, 43.9MB/s]

Downloading (…)l-00004-of-00005.bin:  78%|███████▊  | 3.92G/5.02G [01:04<00:24, 44.3MB/s]

Downloading (…)l-00004-of-00005.bin:  79%|███████▊  | 3.94G/5.02G [01:04<00:17, 60.0MB/s]

Downloading (…)l-00004-of-00005.bin:  79%|███████▉  | 3.95G/5.02G [01:05<00:18, 58.4MB/s]

Downloading (…)l-00004-of-00005.bin:  79%|███████▉  | 3.97G/5.02G [01:05<00:16, 63.7MB/s]

Downloading (…)l-00004-of-00005.bin:  79%|███████▉  | 3.98G/5.02G [01:05<00:17, 60.6MB/s]

Downloading (…)l-00004-of-00005.bin:  80%|███████▉  | 4.01G/5.02G [01:05<00:14, 68.0MB/s]

Downloading (…)l-00004-of-00005.bin:  80%|████████  | 4.02G/5.02G [01:06<00:17, 57.8MB/s]

Downloading (…)l-00004-of-00005.bin:  80%|████████  | 4.04G/5.02G [01:06<00:14, 66.3MB/s]

Downloading (…)l-00004-of-00005.bin:  81%|████████  | 4.06G/5.02G [01:06<00:12, 76.8MB/s]

Downloading (…)l-00004-of-00005.bin:  81%|████████  | 4.07G/5.02G [01:06<00:14, 67.0MB/s]

Downloading (…)l-00004-of-00005.bin:  82%|████████▏ | 4.09G/5.02G [01:07<00:13, 69.2MB/s]

Downloading (…)l-00004-of-00005.bin:  82%|████████▏ | 4.10G/5.02G [01:07<00:13, 68.5MB/s]

Downloading (…)l-00004-of-00005.bin:  82%|████████▏ | 4.12G/5.02G [01:07<00:12, 71.9MB/s]

Downloading (…)l-00004-of-00005.bin:  82%|████████▏ | 4.13G/5.02G [01:07<00:13, 66.1MB/s]

Downloading (…)l-00004-of-00005.bin:  83%|████████▎ | 4.15G/5.02G [01:07<00:10, 79.0MB/s]

Downloading (…)l-00004-of-00005.bin:  83%|████████▎ | 4.16G/5.02G [01:08<00:11, 74.7MB/s]

Downloading (…)l-00004-of-00005.bin:  83%|████████▎ | 4.18G/5.02G [01:08<00:10, 76.8MB/s]

Downloading (…)l-00004-of-00005.bin:  84%|████████▎ | 4.19G/5.02G [01:08<00:12, 64.1MB/s]

Downloading (…)l-00004-of-00005.bin:  84%|████████▍ | 4.22G/5.02G [01:08<00:12, 64.6MB/s]

Downloading (…)l-00004-of-00005.bin:  84%|████████▍ | 4.23G/5.02G [01:09<00:13, 59.2MB/s]

Downloading (…)l-00004-of-00005.bin:  85%|████████▍ | 4.25G/5.02G [01:09<00:14, 53.3MB/s]

Downloading (…)l-00004-of-00005.bin:  85%|████████▍ | 4.26G/5.02G [01:09<00:14, 51.7MB/s]

Downloading (…)l-00004-of-00005.bin:  85%|████████▌ | 4.28G/5.02G [01:10<00:11, 63.4MB/s]

Downloading (…)l-00004-of-00005.bin:  86%|████████▌ | 4.29G/5.02G [01:10<00:12, 60.5MB/s]

Downloading (…)l-00004-of-00005.bin:  86%|████████▌ | 4.30G/5.02G [01:10<00:10, 67.0MB/s]

Downloading (…)l-00004-of-00005.bin:  86%|████████▌ | 4.31G/5.02G [01:10<00:09, 73.2MB/s]

Downloading (…)l-00004-of-00005.bin:  86%|████████▌ | 4.32G/5.02G [01:10<00:09, 73.0MB/s]

Downloading (…)l-00004-of-00005.bin:  87%|████████▋ | 4.34G/5.02G [01:10<00:09, 70.5MB/s]

Downloading (…)l-00004-of-00005.bin:  87%|████████▋ | 4.36G/5.02G [01:11<00:08, 76.6MB/s]

Downloading (…)l-00004-of-00005.bin:  87%|████████▋ | 4.37G/5.02G [01:11<00:09, 66.4MB/s]

Downloading (…)l-00004-of-00005.bin:  88%|████████▊ | 4.39G/5.02G [01:11<00:08, 72.1MB/s]

Downloading (…)l-00004-of-00005.bin:  88%|████████▊ | 4.40G/5.02G [01:11<00:08, 70.6MB/s]

Downloading (…)l-00004-of-00005.bin:  88%|████████▊ | 4.41G/5.02G [01:12<00:10, 58.7MB/s]

Downloading (…)l-00004-of-00005.bin:  88%|████████▊ | 4.42G/5.02G [01:12<00:11, 52.1MB/s]

Downloading (…)l-00004-of-00005.bin:  88%|████████▊ | 4.44G/5.02G [01:12<00:12, 44.7MB/s]

Downloading (…)l-00004-of-00005.bin:  89%|████████▉ | 4.46G/5.02G [01:12<00:09, 60.2MB/s]

Downloading (…)l-00004-of-00005.bin:  89%|████████▉ | 4.47G/5.02G [01:13<00:10, 52.4MB/s]

Downloading (…)l-00004-of-00005.bin:  89%|████████▉ | 4.49G/5.02G [01:13<00:09, 55.3MB/s]

Downloading (…)l-00004-of-00005.bin:  90%|████████▉ | 4.50G/5.02G [01:13<00:10, 48.1MB/s]

Downloading (…)l-00004-of-00005.bin:  90%|█████████ | 4.52G/5.02G [01:14<00:09, 51.6MB/s]

Downloading (…)l-00004-of-00005.bin:  90%|█████████ | 4.53G/5.02G [01:14<00:09, 48.7MB/s]

Downloading (…)l-00004-of-00005.bin:  91%|█████████ | 4.55G/5.02G [01:14<00:08, 57.5MB/s]

Downloading (…)l-00004-of-00005.bin:  91%|█████████ | 4.56G/5.02G [01:14<00:07, 60.7MB/s]

Downloading (…)l-00004-of-00005.bin:  91%|█████████▏| 4.58G/5.02G [01:15<00:05, 74.7MB/s]

Downloading (…)l-00004-of-00005.bin:  92%|█████████▏| 4.59G/5.02G [01:15<00:05, 75.5MB/s]

Downloading (…)l-00004-of-00005.bin:  92%|█████████▏| 4.61G/5.02G [01:15<00:05, 80.1MB/s]

Downloading (…)l-00004-of-00005.bin:  92%|█████████▏| 4.62G/5.02G [01:15<00:06, 60.8MB/s]

Downloading (…)l-00004-of-00005.bin:  93%|█████████▎| 4.65G/5.02G [01:15<00:05, 70.3MB/s]

Downloading (…)l-00004-of-00005.bin:  93%|█████████▎| 4.67G/5.02G [01:16<00:04, 81.7MB/s]

Downloading (…)l-00004-of-00005.bin:  93%|█████████▎| 4.68G/5.02G [01:16<00:04, 71.8MB/s]

Downloading (…)l-00004-of-00005.bin:  94%|█████████▎| 4.70G/5.02G [01:16<00:04, 65.8MB/s]

Downloading (…)l-00004-of-00005.bin:  94%|█████████▍| 4.71G/5.02G [01:16<00:05, 60.5MB/s]

Downloading (…)l-00004-of-00005.bin:  94%|█████████▍| 4.73G/5.02G [01:17<00:04, 66.0MB/s]

Downloading (…)l-00004-of-00005.bin:  95%|█████████▍| 4.74G/5.02G [01:17<00:04, 68.5MB/s]

Downloading (…)l-00004-of-00005.bin:  95%|█████████▍| 4.76G/5.02G [01:17<00:04, 62.3MB/s]

Downloading (…)l-00004-of-00005.bin:  95%|█████████▌| 4.77G/5.02G [01:17<00:04, 55.4MB/s]

Downloading (…)l-00004-of-00005.bin:  95%|█████████▌| 4.78G/5.02G [01:18<00:04, 52.4MB/s]

Downloading (…)l-00004-of-00005.bin:  96%|█████████▌| 4.79G/5.02G [01:18<00:04, 50.6MB/s]

Downloading (…)l-00004-of-00005.bin:  96%|█████████▌| 4.80G/5.02G [01:18<00:03, 53.4MB/s]

Downloading (…)l-00004-of-00005.bin:  96%|█████████▌| 4.82G/5.02G [01:18<00:02, 73.4MB/s]

Downloading (…)l-00004-of-00005.bin:  96%|█████████▋| 4.83G/5.02G [01:19<00:03, 46.8MB/s]

Downloading (…)l-00004-of-00005.bin:  97%|█████████▋| 4.85G/5.02G [01:19<00:03, 52.4MB/s]

Downloading (…)l-00004-of-00005.bin:  97%|█████████▋| 4.87G/5.02G [01:19<00:02, 55.6MB/s]

Downloading (…)l-00004-of-00005.bin:  97%|█████████▋| 4.88G/5.02G [01:19<00:02, 51.3MB/s]

Downloading (…)l-00004-of-00005.bin:  97%|█████████▋| 4.89G/5.02G [01:20<00:02, 55.4MB/s]

Downloading (…)l-00004-of-00005.bin:  98%|█████████▊| 4.90G/5.02G [01:20<00:02, 57.9MB/s]

Downloading (…)l-00004-of-00005.bin:  98%|█████████▊| 4.91G/5.02G [01:20<00:01, 55.6MB/s]

Downloading (…)l-00004-of-00005.bin:  98%|█████████▊| 4.92G/5.02G [01:20<00:01, 53.6MB/s]

Downloading (…)l-00004-of-00005.bin:  98%|█████████▊| 4.93G/5.02G [01:20<00:01, 49.1MB/s]

Downloading (…)l-00004-of-00005.bin:  99%|█████████▊| 4.95G/5.02G [01:21<00:01, 60.0MB/s]

Downloading (…)l-00004-of-00005.bin:  99%|█████████▉| 4.97G/5.02G [01:21<00:00, 78.3MB/s]

Downloading (…)l-00004-of-00005.bin:  99%|█████████▉| 4.98G/5.02G [01:21<00:00, 67.7MB/s]

Downloading (…)l-00004-of-00005.bin: 100%|█████████▉| 5.00G/5.02G [01:21<00:00, 63.2MB/s]

Downloading (…)l-00004-of-00005.bin: 100%|█████████▉| 5.01G/5.02G [01:22<00:00, 65.6MB/s]
Downloading (…)l-00004-of-00005.bin: 100%|██████████| 5.02G/5.02G [01:22<00:00, 61.0MB/s]

Downloading shards:  80%|████████  | 4/5 [05:23<01:20, 80.83s/it]

Downloading (…)l-00005-of-00005.bin:   0%|          | 0.00/4.16G [00:00<?, ?B/s]

Downloading (…)l-00005-of-00005.bin:   0%|          | 10.5M/4.16G [00:00<01:03, 64.9MB/s]

Downloading (…)l-00005-of-00005.bin:   1%|          | 21.0M/4.16G [00:00<01:26, 47.6MB/s]

Downloading (…)l-00005-of-00005.bin:   1%|          | 41.9M/4.16G [00:00<00:56, 72.8MB/s]

Downloading (…)l-00005-of-00005.bin:   1%|▏         | 52.4M/4.16G [00:00<00:58, 70.3MB/s]

Downloading (…)l-00005-of-00005.bin:   2%|▏         | 73.4M/4.16G [00:01<00:55, 73.2MB/s]

Downloading (…)l-00005-of-00005.bin:   2%|▏         | 83.9M/4.16G [00:01<00:56, 72.6MB/s]

Downloading (…)l-00005-of-00005.bin:   3%|▎         | 105M/4.16G [00:01<01:01, 66.4MB/s] 

Downloading (…)l-00005-of-00005.bin:   3%|▎         | 115M/4.16G [00:01<01:13, 55.2MB/s]

Downloading (…)l-00005-of-00005.bin:   3%|▎         | 136M/4.16G [00:02<00:58, 68.4MB/s]

Downloading (…)l-00005-of-00005.bin:   4%|▎         | 147M/4.16G [00:02<01:05, 60.9MB/s]

Downloading (…)l-00005-of-00005.bin:   4%|▍         | 168M/4.16G [00:02<00:58, 68.7MB/s]

Downloading (…)l-00005-of-00005.bin:   4%|▍         | 178M/4.16G [00:02<01:05, 60.6MB/s]

Downloading (…)l-00005-of-00005.bin:   5%|▍         | 199M/4.16G [00:03<01:03, 62.6MB/s]

Downloading (…)l-00005-of-00005.bin:   5%|▌         | 210M/4.16G [00:03<01:14, 52.7MB/s]

Downloading (…)l-00005-of-00005.bin:   6%|▌         | 231M/4.16G [00:03<00:58, 66.6MB/s]

Downloading (…)l-00005-of-00005.bin:   6%|▌         | 241M/4.16G [00:03<00:59, 65.8MB/s]

Downloading (…)l-00005-of-00005.bin:   6%|▋         | 262M/4.16G [00:04<01:00, 64.0MB/s]

Downloading (…)l-00005-of-00005.bin:   7%|▋         | 273M/4.16G [00:04<00:58, 66.2MB/s]

Downloading (…)l-00005-of-00005.bin:   7%|▋         | 294M/4.16G [00:04<00:53, 71.7MB/s]

Downloading (…)l-00005-of-00005.bin:   7%|▋         | 304M/4.16G [00:04<00:58, 66.3MB/s]

Downloading (…)l-00005-of-00005.bin:   8%|▊         | 315M/4.16G [00:04<01:00, 63.1MB/s]

Downloading (…)l-00005-of-00005.bin:   8%|▊         | 325M/4.16G [00:05<01:08, 55.6MB/s]

Downloading (…)l-00005-of-00005.bin:   8%|▊         | 336M/4.16G [00:05<01:03, 59.7MB/s]

Downloading (…)l-00005-of-00005.bin:   8%|▊         | 346M/4.16G [00:05<01:08, 56.0MB/s]

Downloading (…)l-00005-of-00005.bin:   9%|▊         | 357M/4.16G [00:05<01:05, 58.3MB/s]

Downloading (…)l-00005-of-00005.bin:   9%|▉         | 377M/4.16G [00:06<01:06, 56.5MB/s]

Downloading (…)l-00005-of-00005.bin:   9%|▉         | 388M/4.16G [00:06<01:02, 60.6MB/s]

Downloading (…)l-00005-of-00005.bin:  10%|▉         | 409M/4.16G [00:06<00:59, 63.3MB/s]

Downloading (…)l-00005-of-00005.bin:  10%|█         | 419M/4.16G [00:06<00:57, 65.4MB/s]

Downloading (…)l-00005-of-00005.bin:  11%|█         | 440M/4.16G [00:06<00:56, 65.8MB/s]

Downloading (…)l-00005-of-00005.bin:  11%|█         | 451M/4.16G [00:07<01:05, 56.4MB/s]

Downloading (…)l-00005-of-00005.bin:  11%|█▏        | 472M/4.16G [00:07<00:58, 63.1MB/s]

Downloading (…)l-00005-of-00005.bin:  12%|█▏        | 482M/4.16G [00:07<01:00, 60.8MB/s]

Downloading (…)l-00005-of-00005.bin:  12%|█▏        | 503M/4.16G [00:07<00:51, 71.5MB/s]

Downloading (…)l-00005-of-00005.bin:  12%|█▏        | 514M/4.16G [00:08<00:52, 69.8MB/s]

Downloading (…)l-00005-of-00005.bin:  13%|█▎        | 535M/4.16G [00:08<00:46, 78.0MB/s]

Downloading (…)l-00005-of-00005.bin:  13%|█▎        | 545M/4.16G [00:08<00:55, 64.6MB/s]

Downloading (…)l-00005-of-00005.bin:  13%|█▎        | 556M/4.16G [00:08<00:53, 67.5MB/s]

Downloading (…)l-00005-of-00005.bin:  14%|█▎        | 566M/4.16G [00:08<00:52, 68.2MB/s]

Downloading (…)l-00005-of-00005.bin:  14%|█▍        | 577M/4.16G [00:08<00:51, 69.3MB/s]

Downloading (…)l-00005-of-00005.bin:  14%|█▍        | 598M/4.16G [00:09<00:55, 64.3MB/s]

Downloading (…)l-00005-of-00005.bin:  15%|█▍        | 608M/4.16G [00:09<00:57, 61.8MB/s]

Downloading (…)l-00005-of-00005.bin:  15%|█▌        | 629M/4.16G [00:09<00:47, 73.8MB/s]

Downloading (…)l-00005-of-00005.bin:  15%|█▌        | 640M/4.16G [00:09<00:46, 75.6MB/s]

Downloading (…)l-00005-of-00005.bin:  16%|█▌        | 650M/4.16G [00:10<00:52, 67.1MB/s]

Downloading (…)l-00005-of-00005.bin:  16%|█▌        | 661M/4.16G [00:10<00:53, 65.4MB/s]

Downloading (…)l-00005-of-00005.bin:  16%|█▌        | 671M/4.16G [00:10<00:48, 72.4MB/s]

Downloading (…)l-00005-of-00005.bin:  16%|█▋        | 682M/4.16G [00:10<01:04, 54.1MB/s]

Downloading (…)l-00005-of-00005.bin:  17%|█▋        | 692M/4.16G [00:10<01:01, 56.0MB/s]

Downloading (…)l-00005-of-00005.bin:  17%|█▋        | 713M/4.16G [00:11<00:48, 71.5MB/s]

Downloading (…)l-00005-of-00005.bin:  17%|█▋        | 724M/4.16G [00:11<01:02, 55.0MB/s]

Downloading (…)l-00005-of-00005.bin:  18%|█▊        | 744M/4.16G [00:11<00:57, 59.0MB/s]

Downloading (…)l-00005-of-00005.bin:  18%|█▊        | 755M/4.16G [00:11<01:03, 53.5MB/s]

Downloading (…)l-00005-of-00005.bin:  19%|█▊        | 776M/4.16G [00:12<00:49, 67.8MB/s]

Downloading (…)l-00005-of-00005.bin:  19%|█▉        | 786M/4.16G [00:12<00:53, 62.7MB/s]

Downloading (…)l-00005-of-00005.bin:  19%|█▉        | 807M/4.16G [00:12<00:50, 66.8MB/s]

Downloading (…)l-00005-of-00005.bin:  20%|█▉        | 818M/4.16G [00:12<00:49, 67.4MB/s]

Downloading (…)l-00005-of-00005.bin:  20%|██        | 839M/4.16G [00:12<00:40, 81.3MB/s]

Downloading (…)l-00005-of-00005.bin:  20%|██        | 849M/4.16G [00:13<00:44, 75.1MB/s]

Downloading (…)l-00005-of-00005.bin:  21%|██        | 870M/4.16G [00:13<00:39, 82.4MB/s]

Downloading (…)l-00005-of-00005.bin:  21%|██        | 881M/4.16G [00:13<00:41, 79.3MB/s]

Downloading (…)l-00005-of-00005.bin:  22%|██▏       | 902M/4.16G [00:13<00:44, 72.9MB/s]

Downloading (…)l-00005-of-00005.bin:  22%|██▏       | 912M/4.16G [00:14<00:59, 54.5MB/s]

Downloading (…)l-00005-of-00005.bin:  22%|██▏       | 933M/4.16G [00:14<00:50, 64.2MB/s]

Downloading (…)l-00005-of-00005.bin:  23%|██▎       | 954M/4.16G [00:14<00:47, 67.5MB/s]

Downloading (…)l-00005-of-00005.bin:  23%|██▎       | 965M/4.16G [00:14<00:50, 63.3MB/s]

Downloading (…)l-00005-of-00005.bin:  23%|██▎       | 975M/4.16G [00:15<00:48, 65.9MB/s]

Downloading (…)l-00005-of-00005.bin:  24%|██▎       | 986M/4.16G [00:15<00:57, 55.5MB/s]

Downloading (…)l-00005-of-00005.bin:  24%|██▍       | 996M/4.16G [00:15<01:01, 51.5MB/s]

Downloading (…)l-00005-of-00005.bin:  24%|██▍       | 1.02G/4.16G [00:15<00:56, 55.2MB/s]

Downloading (…)l-00005-of-00005.bin:  25%|██▍       | 1.03G/4.16G [00:16<00:54, 57.1MB/s]

Downloading (…)l-00005-of-00005.bin:  25%|██▍       | 1.04G/4.16G [00:16<00:51, 61.0MB/s]

Downloading (…)l-00005-of-00005.bin:  25%|██▌       | 1.05G/4.16G [00:16<01:08, 45.1MB/s]

Downloading (…)l-00005-of-00005.bin:  25%|██▌       | 1.06G/4.16G [00:16<01:08, 45.4MB/s]

Downloading (…)l-00005-of-00005.bin:  26%|██▌       | 1.08G/4.16G [00:17<00:58, 52.3MB/s]

Downloading (…)l-00005-of-00005.bin:  26%|██▌       | 1.09G/4.16G [00:17<00:56, 54.2MB/s]

Downloading (…)l-00005-of-00005.bin:  27%|██▋       | 1.11G/4.16G [00:17<00:47, 64.0MB/s]

Downloading (…)l-00005-of-00005.bin:  27%|██▋       | 1.12G/4.16G [00:17<00:46, 65.3MB/s]

Downloading (…)l-00005-of-00005.bin:  27%|██▋       | 1.14G/4.16G [00:18<00:46, 65.5MB/s]

Downloading (…)l-00005-of-00005.bin:  28%|██▊       | 1.15G/4.16G [00:18<00:51, 58.4MB/s]

Downloading (…)l-00005-of-00005.bin:  28%|██▊       | 1.17G/4.16G [00:18<00:41, 71.5MB/s]

Downloading (…)l-00005-of-00005.bin:  28%|██▊       | 1.18G/4.16G [00:18<00:41, 70.8MB/s]

Downloading (…)l-00005-of-00005.bin:  29%|██▉       | 1.21G/4.16G [00:18<00:37, 79.4MB/s]

Downloading (…)l-00005-of-00005.bin:  29%|██▉       | 1.22G/4.16G [00:19<00:40, 71.9MB/s]

Downloading (…)l-00005-of-00005.bin:  30%|██▉       | 1.24G/4.16G [00:19<00:41, 70.9MB/s]

Downloading (…)l-00005-of-00005.bin:  30%|███       | 1.25G/4.16G [00:19<00:43, 67.0MB/s]

Downloading (…)l-00005-of-00005.bin:  30%|███       | 1.26G/4.16G [00:19<00:43, 67.1MB/s]

Downloading (…)l-00005-of-00005.bin:  31%|███       | 1.27G/4.16G [00:19<00:52, 54.6MB/s]

Downloading (…)l-00005-of-00005.bin:  31%|███       | 1.28G/4.16G [00:20<00:58, 48.9MB/s]

Downloading (…)l-00005-of-00005.bin:  31%|███       | 1.29G/4.16G [00:20<00:55, 51.9MB/s]

Downloading (…)l-00005-of-00005.bin:  31%|███▏      | 1.30G/4.16G [00:20<00:54, 52.7MB/s]

Downloading (…)l-00005-of-00005.bin:  32%|███▏      | 1.32G/4.16G [00:20<00:50, 56.1MB/s]

Downloading (…)l-00005-of-00005.bin:  32%|███▏      | 1.33G/4.16G [00:21<00:53, 52.9MB/s]

Downloading (…)l-00005-of-00005.bin:  33%|███▎      | 1.35G/4.16G [00:21<00:46, 60.6MB/s]

Downloading (…)l-00005-of-00005.bin:  33%|███▎      | 1.36G/4.16G [00:21<00:45, 61.1MB/s]

Downloading (…)l-00005-of-00005.bin:  33%|███▎      | 1.37G/4.16G [00:22<01:57, 23.7MB/s]

Downloading (…)l-00005-of-00005.bin:  33%|███▎      | 1.38G/4.16G [00:23<01:41, 27.2MB/s]

Downloading (…)l-00005-of-00005.bin:  34%|███▎      | 1.39G/4.16G [00:23<01:28, 31.3MB/s]

Downloading (…)l-00005-of-00005.bin:  34%|███▍      | 1.42G/4.16G [00:23<01:09, 39.7MB/s]

Downloading (…)l-00005-of-00005.bin:  34%|███▍      | 1.43G/4.16G [00:23<01:05, 41.5MB/s]

Downloading (…)l-00005-of-00005.bin:  35%|███▍      | 1.45G/4.16G [00:24<00:50, 53.2MB/s]

Downloading (…)l-00005-of-00005.bin:  35%|███▌      | 1.46G/4.16G [00:24<00:48, 55.7MB/s]

Downloading (…)l-00005-of-00005.bin:  36%|███▌      | 1.48G/4.16G [00:24<00:40, 66.7MB/s]

Downloading (…)l-00005-of-00005.bin:  36%|███▌      | 1.49G/4.16G [00:24<00:43, 61.6MB/s]

Downloading (…)l-00005-of-00005.bin:  36%|███▋      | 1.51G/4.16G [00:24<00:36, 71.8MB/s]

Downloading (…)l-00005-of-00005.bin:  37%|███▋      | 1.52G/4.16G [00:25<00:43, 61.3MB/s]

Downloading (…)l-00005-of-00005.bin:  37%|███▋      | 1.54G/4.16G [00:25<00:35, 74.3MB/s]

Downloading (…)l-00005-of-00005.bin:  38%|███▊      | 1.56G/4.16G [00:25<00:34, 75.1MB/s]

Downloading (…)l-00005-of-00005.bin:  38%|███▊      | 1.57G/4.16G [00:25<00:40, 64.5MB/s]

Downloading (…)l-00005-of-00005.bin:  38%|███▊      | 1.59G/4.16G [00:26<00:35, 71.3MB/s]

Downloading (…)l-00005-of-00005.bin:  39%|███▊      | 1.60G/4.16G [00:26<00:34, 74.4MB/s]

Downloading (…)l-00005-of-00005.bin:  39%|███▉      | 1.63G/4.16G [00:26<00:38, 66.6MB/s]

Downloading (…)l-00005-of-00005.bin:  39%|███▉      | 1.64G/4.16G [00:26<00:37, 67.4MB/s]

Downloading (…)l-00005-of-00005.bin:  40%|███▉      | 1.65G/4.16G [00:26<00:38, 65.1MB/s]

Downloading (…)l-00005-of-00005.bin:  40%|███▉      | 1.66G/4.16G [00:27<00:40, 62.5MB/s]

Downloading (…)l-00005-of-00005.bin:  40%|████      | 1.67G/4.16G [00:27<00:43, 57.9MB/s]

Downloading (…)l-00005-of-00005.bin:  41%|████      | 1.69G/4.16G [00:27<00:40, 60.5MB/s]

Downloading (…)l-00005-of-00005.bin:  41%|████      | 1.70G/4.16G [00:27<00:48, 51.2MB/s]

Downloading (…)l-00005-of-00005.bin:  41%|████▏     | 1.72G/4.16G [00:28<00:41, 58.7MB/s]

Downloading (…)l-00005-of-00005.bin:  42%|████▏     | 1.73G/4.16G [00:28<00:39, 61.1MB/s]

Downloading (…)l-00005-of-00005.bin:  42%|████▏     | 1.75G/4.16G [00:28<00:39, 61.7MB/s]

Downloading (…)l-00005-of-00005.bin:  42%|████▏     | 1.76G/4.16G [00:28<00:38, 62.7MB/s]

Downloading (…)l-00005-of-00005.bin:  43%|████▎     | 1.78G/4.16G [00:29<00:32, 73.3MB/s]

Downloading (…)l-00005-of-00005.bin:  43%|████▎     | 1.79G/4.16G [00:29<00:36, 64.1MB/s]

Downloading (…)l-00005-of-00005.bin:  44%|████▎     | 1.81G/4.16G [00:29<00:34, 67.7MB/s]

Downloading (…)l-00005-of-00005.bin:  44%|████▍     | 1.82G/4.16G [00:29<00:36, 64.6MB/s]

Downloading (…)l-00005-of-00005.bin:  44%|████▍     | 1.85G/4.16G [00:30<00:35, 65.3MB/s]

Downloading (…)l-00005-of-00005.bin:  45%|████▍     | 1.86G/4.16G [00:30<00:33, 69.4MB/s]

Downloading (…)l-00005-of-00005.bin:  45%|████▍     | 1.87G/4.16G [00:30<00:33, 67.8MB/s]

Downloading (…)l-00005-of-00005.bin:  45%|████▌     | 1.88G/4.16G [00:30<00:39, 57.2MB/s]

Downloading (…)l-00005-of-00005.bin:  46%|████▌     | 1.90G/4.16G [00:30<00:33, 67.7MB/s]

Downloading (…)l-00005-of-00005.bin:  46%|████▌     | 1.91G/4.16G [00:31<00:37, 60.6MB/s]

Downloading (…)l-00005-of-00005.bin:  46%|████▋     | 1.93G/4.16G [00:31<00:34, 65.5MB/s]

Downloading (…)l-00005-of-00005.bin:  47%|████▋     | 1.94G/4.16G [00:31<00:39, 55.8MB/s]

Downloading (…)l-00005-of-00005.bin:  47%|████▋     | 1.96G/4.16G [00:31<00:31, 69.3MB/s]

Downloading (…)l-00005-of-00005.bin:  47%|████▋     | 1.97G/4.16G [00:32<00:36, 59.8MB/s]

Downloading (…)l-00005-of-00005.bin:  48%|████▊     | 1.99G/4.16G [00:32<00:32, 66.4MB/s]

Downloading (…)l-00005-of-00005.bin:  48%|████▊     | 2.00G/4.16G [00:32<00:33, 64.6MB/s]

Downloading (…)l-00005-of-00005.bin:  49%|████▊     | 2.02G/4.16G [00:33<00:38, 55.7MB/s]

Downloading (…)l-00005-of-00005.bin:  49%|████▉     | 2.03G/4.16G [00:33<00:40, 52.8MB/s]

Downloading (…)l-00005-of-00005.bin:  49%|████▉     | 2.06G/4.16G [00:33<00:40, 51.5MB/s]

Downloading (…)l-00005-of-00005.bin:  50%|████▉     | 2.07G/4.16G [00:33<00:37, 55.7MB/s]

Downloading (…)l-00005-of-00005.bin:  50%|█████     | 2.09G/4.16G [00:34<00:31, 66.6MB/s]

Downloading (…)l-00005-of-00005.bin:  50%|█████     | 2.10G/4.16G [00:34<00:31, 64.7MB/s]

Downloading (…)l-00005-of-00005.bin:  51%|█████     | 2.11G/4.16G [00:34<00:33, 61.5MB/s]

Downloading (…)l-00005-of-00005.bin:  51%|█████     | 2.12G/4.16G [00:34<00:31, 65.6MB/s]

Downloading (…)l-00005-of-00005.bin:  51%|█████     | 2.13G/4.16G [00:34<00:42, 47.2MB/s]

Downloading (…)l-00005-of-00005.bin:  52%|█████▏    | 2.15G/4.16G [00:35<00:36, 54.6MB/s]

Downloading (…)l-00005-of-00005.bin:  52%|█████▏    | 2.16G/4.16G [00:35<00:41, 47.7MB/s]

Downloading (…)l-00005-of-00005.bin:  52%|█████▏    | 2.18G/4.16G [00:35<00:30, 65.1MB/s]

Downloading (…)l-00005-of-00005.bin:  53%|█████▎    | 2.20G/4.16G [00:35<00:25, 76.0MB/s]

Downloading (…)l-00005-of-00005.bin:  53%|█████▎    | 2.21G/4.16G [00:36<00:26, 73.7MB/s]

Downloading (…)l-00005-of-00005.bin:  54%|█████▎    | 2.23G/4.16G [00:36<00:26, 71.5MB/s]

Downloading (…)l-00005-of-00005.bin:  54%|█████▍    | 2.24G/4.16G [00:36<00:27, 70.8MB/s]

Downloading (…)l-00005-of-00005.bin:  54%|█████▍    | 2.26G/4.16G [00:36<00:26, 70.8MB/s]

Downloading (…)l-00005-of-00005.bin:  55%|█████▍    | 2.28G/4.16G [00:37<00:29, 63.2MB/s]

Downloading (…)l-00005-of-00005.bin:  55%|█████▌    | 2.30G/4.16G [00:37<00:27, 67.4MB/s]

Downloading (…)l-00005-of-00005.bin:  55%|█████▌    | 2.31G/4.16G [00:37<00:31, 59.6MB/s]

Downloading (…)l-00005-of-00005.bin:  56%|█████▌    | 2.32G/4.16G [00:37<00:33, 54.8MB/s]

Downloading (…)l-00005-of-00005.bin:  56%|█████▌    | 2.33G/4.16G [00:38<00:32, 55.6MB/s]

Downloading (…)l-00005-of-00005.bin:  56%|█████▌    | 2.34G/4.16G [00:38<00:31, 58.1MB/s]

Downloading (…)l-00005-of-00005.bin:  57%|█████▋    | 2.36G/4.16G [00:38<00:26, 69.0MB/s]

Downloading (…)l-00005-of-00005.bin:  57%|█████▋    | 2.37G/4.16G [00:38<00:29, 60.5MB/s]

Downloading (…)l-00005-of-00005.bin:  57%|█████▋    | 2.39G/4.16G [00:38<00:23, 75.9MB/s]

Downloading (…)l-00005-of-00005.bin:  58%|█████▊    | 2.40G/4.16G [00:39<00:24, 72.3MB/s]

Downloading (…)l-00005-of-00005.bin:  58%|█████▊    | 2.41G/4.16G [00:39<00:25, 69.1MB/s]

Downloading (…)l-00005-of-00005.bin:  58%|█████▊    | 2.42G/4.16G [00:39<00:29, 59.5MB/s]

Downloading (…)l-00005-of-00005.bin:  59%|█████▊    | 2.43G/4.16G [00:39<00:29, 58.8MB/s]

Downloading (…)l-00005-of-00005.bin:  59%|█████▉    | 2.45G/4.16G [00:39<00:28, 60.4MB/s]

Downloading (…)l-00005-of-00005.bin:  59%|█████▉    | 2.46G/4.16G [00:40<00:30, 56.0MB/s]

Downloading (…)l-00005-of-00005.bin:  60%|█████▉    | 2.49G/4.16G [00:40<00:31, 53.0MB/s]

Downloading (…)l-00005-of-00005.bin:  60%|██████    | 2.51G/4.16G [00:40<00:26, 61.4MB/s]

Downloading (…)l-00005-of-00005.bin:  61%|██████    | 2.52G/4.16G [00:41<00:28, 57.5MB/s]

Downloading (…)l-00005-of-00005.bin:  61%|██████    | 2.54G/4.16G [00:41<00:22, 72.2MB/s]

Downloading (…)l-00005-of-00005.bin:  61%|██████▏   | 2.55G/4.16G [00:41<00:23, 68.1MB/s]

Downloading (…)l-00005-of-00005.bin:  62%|██████▏   | 2.57G/4.16G [00:41<00:21, 73.5MB/s]

Downloading (…)l-00005-of-00005.bin:  62%|██████▏   | 2.58G/4.16G [00:41<00:20, 77.7MB/s]

Downloading (…)l-00005-of-00005.bin:  63%|██████▎   | 2.60G/4.16G [00:42<00:20, 76.0MB/s]

Downloading (…)l-00005-of-00005.bin:  63%|██████▎   | 2.61G/4.16G [00:42<00:25, 60.4MB/s]

Downloading (…)l-00005-of-00005.bin:  63%|██████▎   | 2.63G/4.16G [00:42<00:28, 54.1MB/s]

Downloading (…)l-00005-of-00005.bin:  64%|██████▎   | 2.64G/4.16G [00:42<00:26, 57.8MB/s]

Downloading (…)l-00005-of-00005.bin:  64%|██████▍   | 2.66G/4.16G [00:43<00:24, 61.1MB/s]

Downloading (…)l-00005-of-00005.bin:  64%|██████▍   | 2.67G/4.16G [00:43<00:25, 59.2MB/s]

Downloading (…)l-00005-of-00005.bin:  65%|██████▍   | 2.69G/4.16G [00:43<00:23, 62.6MB/s]

Downloading (…)l-00005-of-00005.bin:  65%|██████▌   | 2.71G/4.16G [00:43<00:24, 58.9MB/s]

Downloading (…)l-00005-of-00005.bin:  66%|██████▌   | 2.73G/4.16G [00:44<00:21, 67.2MB/s]

Downloading (…)l-00005-of-00005.bin:  66%|██████▌   | 2.74G/4.16G [00:44<00:20, 67.9MB/s]

Downloading (…)l-00005-of-00005.bin:  66%|██████▋   | 2.76G/4.16G [00:44<00:18, 76.3MB/s]

Downloading (…)l-00005-of-00005.bin:  67%|██████▋   | 2.77G/4.16G [00:44<00:18, 74.0MB/s]

Downloading (…)l-00005-of-00005.bin:  67%|██████▋   | 2.78G/4.16G [00:44<00:18, 73.7MB/s]

Downloading (…)l-00005-of-00005.bin:  67%|██████▋   | 2.79G/4.16G [00:45<00:19, 69.1MB/s]

Downloading (…)l-00005-of-00005.bin:  67%|██████▋   | 2.80G/4.16G [00:45<00:18, 74.7MB/s]

Downloading (…)l-00005-of-00005.bin:  68%|██████▊   | 2.81G/4.16G [00:45<00:18, 74.5MB/s]

Downloading (…)l-00005-of-00005.bin:  68%|██████▊   | 2.82G/4.16G [00:45<00:21, 61.0MB/s]

Downloading (…)l-00005-of-00005.bin:  68%|██████▊   | 2.84G/4.16G [00:45<00:18, 69.6MB/s]

Downloading (…)l-00005-of-00005.bin:  69%|██████▊   | 2.85G/4.16G [00:46<00:20, 63.9MB/s]

Downloading (…)l-00005-of-00005.bin:  69%|██████▉   | 2.87G/4.16G [00:46<00:17, 75.6MB/s]

Downloading (…)l-00005-of-00005.bin:  69%|██████▉   | 2.88G/4.16G [00:46<00:16, 76.6MB/s]

Downloading (…)l-00005-of-00005.bin:  70%|██████▉   | 2.90G/4.16G [00:46<00:13, 89.9MB/s]

Downloading (…)l-00005-of-00005.bin:  70%|███████   | 2.92G/4.16G [00:46<00:16, 75.8MB/s]

Downloading (…)l-00005-of-00005.bin:  71%|███████   | 2.94G/4.16G [00:47<00:18, 65.2MB/s]

Downloading (…)l-00005-of-00005.bin:  71%|███████   | 2.95G/4.16G [00:47<00:18, 65.5MB/s]

Downloading (…)l-00005-of-00005.bin:  71%|███████▏  | 2.97G/4.16G [00:47<00:20, 58.0MB/s]

Downloading (…)l-00005-of-00005.bin:  72%|███████▏  | 2.98G/4.16G [00:47<00:20, 58.1MB/s]

Downloading (…)l-00005-of-00005.bin:  72%|███████▏  | 3.00G/4.16G [00:48<00:16, 69.3MB/s]

Downloading (…)l-00005-of-00005.bin:  72%|███████▏  | 3.01G/4.16G [00:48<00:17, 64.2MB/s]

Downloading (…)l-00005-of-00005.bin:  73%|███████▎  | 3.02G/4.16G [00:48<00:18, 62.5MB/s]

Downloading (…)l-00005-of-00005.bin:  73%|███████▎  | 3.03G/4.16G [00:48<00:18, 60.2MB/s]

Downloading (…)l-00005-of-00005.bin:  73%|███████▎  | 3.04G/4.16G [00:48<00:20, 55.3MB/s]

Downloading (…)l-00005-of-00005.bin:  73%|███████▎  | 3.05G/4.16G [00:49<00:18, 59.0MB/s]

Downloading (…)l-00005-of-00005.bin:  74%|███████▎  | 3.06G/4.16G [00:49<00:19, 55.2MB/s]

Downloading (…)l-00005-of-00005.bin:  74%|███████▍  | 3.07G/4.16G [00:49<00:23, 45.4MB/s]

Downloading (…)l-00005-of-00005.bin:  74%|███████▍  | 3.09G/4.16G [00:49<00:20, 52.7MB/s]

Downloading (…)l-00005-of-00005.bin:  75%|███████▍  | 3.11G/4.16G [00:50<00:18, 55.0MB/s]

Downloading (…)l-00005-of-00005.bin:  75%|███████▌  | 3.12G/4.16G [00:50<00:19, 53.7MB/s]

Downloading (…)l-00005-of-00005.bin:  76%|███████▌  | 3.15G/4.16G [00:50<00:19, 52.1MB/s]

Downloading (…)l-00005-of-00005.bin:  76%|███████▌  | 3.16G/4.16G [00:51<00:17, 57.7MB/s]

Downloading (…)l-00005-of-00005.bin:  76%|███████▋  | 3.18G/4.16G [00:51<00:16, 61.0MB/s]

Downloading (…)l-00005-of-00005.bin:  77%|███████▋  | 3.19G/4.16G [00:51<00:15, 60.7MB/s]

Downloading (…)l-00005-of-00005.bin:  77%|███████▋  | 3.21G/4.16G [00:51<00:16, 57.5MB/s]

Downloading (…)l-00005-of-00005.bin:  77%|███████▋  | 3.22G/4.16G [00:52<00:20, 46.1MB/s]

Downloading (…)l-00005-of-00005.bin:  78%|███████▊  | 3.24G/4.16G [00:52<00:18, 50.6MB/s]

Downloading (…)l-00005-of-00005.bin:  78%|███████▊  | 3.25G/4.16G [00:53<00:20, 44.1MB/s]

Downloading (…)l-00005-of-00005.bin:  79%|███████▊  | 3.27G/4.16G [00:53<00:16, 53.9MB/s]

Downloading (…)l-00005-of-00005.bin:  79%|███████▉  | 3.28G/4.16G [00:53<00:19, 44.4MB/s]

Downloading (…)l-00005-of-00005.bin:  79%|███████▉  | 3.30G/4.16G [00:53<00:16, 53.5MB/s]

Downloading (…)l-00005-of-00005.bin:  80%|███████▉  | 3.31G/4.16G [00:54<00:15, 55.6MB/s]

Downloading (…)l-00005-of-00005.bin:  80%|████████  | 3.33G/4.16G [00:54<00:13, 61.2MB/s]

Downloading (…)l-00005-of-00005.bin:  80%|████████  | 3.34G/4.16G [00:54<00:14, 56.1MB/s]

Downloading (…)l-00005-of-00005.bin:  81%|████████  | 3.36G/4.16G [00:54<00:13, 60.9MB/s]

Downloading (…)l-00005-of-00005.bin:  81%|████████  | 3.37G/4.16G [00:54<00:13, 59.6MB/s]

Downloading (…)l-00005-of-00005.bin:  81%|████████  | 3.38G/4.16G [00:55<00:15, 49.2MB/s]

Downloading (…)l-00005-of-00005.bin:  82%|████████▏ | 3.40G/4.16G [00:55<00:14, 53.7MB/s]

Downloading (…)l-00005-of-00005.bin:  82%|████████▏ | 3.42G/4.16G [00:55<00:11, 63.8MB/s]

Downloading (…)l-00005-of-00005.bin:  82%|████████▏ | 3.43G/4.16G [00:56<00:12, 59.7MB/s]

Downloading (…)l-00005-of-00005.bin:  83%|████████▎ | 3.45G/4.16G [00:56<00:12, 58.8MB/s]

Downloading (…)l-00005-of-00005.bin:  83%|████████▎ | 3.46G/4.16G [00:56<00:11, 58.9MB/s]

Downloading (…)l-00005-of-00005.bin:  84%|████████▎ | 3.48G/4.16G [00:56<00:09, 72.1MB/s]

Downloading (…)l-00005-of-00005.bin:  84%|████████▍ | 3.49G/4.16G [00:57<00:10, 60.6MB/s]

Downloading (…)l-00005-of-00005.bin:  84%|████████▍ | 3.51G/4.16G [00:57<00:10, 60.0MB/s]

Downloading (…)l-00005-of-00005.bin:  85%|████████▍ | 3.52G/4.16G [00:57<00:10, 58.7MB/s]

Downloading (…)l-00005-of-00005.bin:  85%|████████▌ | 3.54G/4.16G [00:57<00:09, 68.2MB/s]

Downloading (…)l-00005-of-00005.bin:  85%|████████▌ | 3.55G/4.16G [00:57<00:09, 66.9MB/s]

Downloading (…)l-00005-of-00005.bin:  86%|████████▌ | 3.58G/4.16G [00:58<00:07, 74.6MB/s]

Downloading (…)l-00005-of-00005.bin:  86%|████████▌ | 3.59G/4.16G [00:58<00:08, 68.9MB/s]

Downloading (…)l-00005-of-00005.bin:  86%|████████▋ | 3.60G/4.16G [00:58<00:10, 53.2MB/s]

Downloading (…)l-00005-of-00005.bin:  87%|████████▋ | 3.61G/4.16G [00:58<00:10, 52.7MB/s]

Downloading (…)l-00005-of-00005.bin:  87%|████████▋ | 3.62G/4.16G [00:59<00:09, 57.0MB/s]

Downloading (…)l-00005-of-00005.bin:  87%|████████▋ | 3.64G/4.16G [00:59<00:08, 63.6MB/s]

Downloading (…)l-00005-of-00005.bin:  88%|████████▊ | 3.65G/4.16G [00:59<00:08, 62.7MB/s]

Downloading (…)l-00005-of-00005.bin:  88%|████████▊ | 3.67G/4.16G [00:59<00:06, 70.0MB/s]

Downloading (…)l-00005-of-00005.bin:  89%|████████▊ | 3.68G/4.16G [01:00<00:09, 50.4MB/s]

Downloading (…)l-00005-of-00005.bin:  89%|████████▉ | 3.70G/4.16G [01:00<00:08, 54.3MB/s]

Downloading (…)l-00005-of-00005.bin:  89%|████████▉ | 3.71G/4.16G [01:00<00:07, 57.3MB/s]

Downloading (…)l-00005-of-00005.bin:  90%|████████▉ | 3.72G/4.16G [01:00<00:07, 57.1MB/s]

Downloading (…)l-00005-of-00005.bin:  90%|████████▉ | 3.73G/4.16G [01:01<00:08, 47.4MB/s]

Downloading (…)l-00005-of-00005.bin:  90%|█████████ | 3.75G/4.16G [01:01<00:06, 62.6MB/s]

Downloading (…)l-00005-of-00005.bin:  91%|█████████ | 3.76G/4.16G [01:01<00:06, 62.1MB/s]

Downloading (…)l-00005-of-00005.bin:  91%|█████████ | 3.77G/4.16G [01:01<00:06, 57.9MB/s]

Downloading (…)l-00005-of-00005.bin:  91%|█████████ | 3.79G/4.16G [01:01<00:06, 58.4MB/s]

Downloading (…)l-00005-of-00005.bin:  91%|█████████▏| 3.80G/4.16G [01:02<00:07, 50.3MB/s]

Downloading (…)l-00005-of-00005.bin:  92%|█████████▏| 3.82G/4.16G [01:02<00:06, 53.1MB/s]

Downloading (…)l-00005-of-00005.bin:  92%|█████████▏| 3.83G/4.16G [01:02<00:06, 49.5MB/s]

Downloading (…)l-00005-of-00005.bin:  93%|█████████▎| 3.85G/4.16G [01:03<00:06, 51.4MB/s]

Downloading (…)l-00005-of-00005.bin:  93%|█████████▎| 3.86G/4.16G [01:03<00:06, 45.6MB/s]

Downloading (…)l-00005-of-00005.bin:  93%|█████████▎| 3.87G/4.16G [01:03<00:05, 51.7MB/s]

Downloading (…)l-00005-of-00005.bin:  93%|█████████▎| 3.88G/4.16G [01:04<00:06, 46.1MB/s]

Downloading (…)l-00005-of-00005.bin:  94%|█████████▎| 3.89G/4.16G [01:04<00:05, 46.5MB/s]

Downloading (…)l-00005-of-00005.bin:  94%|█████████▍| 3.91G/4.16G [01:04<00:04, 52.5MB/s]

Downloading (…)l-00005-of-00005.bin:  94%|█████████▍| 3.92G/4.16G [01:04<00:04, 55.7MB/s]

Downloading (…)l-00005-of-00005.bin:  95%|█████████▍| 3.93G/4.16G [01:04<00:03, 59.9MB/s]

Downloading (…)l-00005-of-00005.bin:  95%|█████████▍| 3.94G/4.16G [01:04<00:03, 63.2MB/s]

Downloading (…)l-00005-of-00005.bin:  95%|█████████▌| 3.95G/4.16G [01:05<00:03, 58.8MB/s]

Downloading (…)l-00005-of-00005.bin:  96%|█████████▌| 3.97G/4.16G [01:05<00:02, 66.7MB/s]

Downloading (…)l-00005-of-00005.bin:  96%|█████████▌| 3.98G/4.16G [01:05<00:02, 59.1MB/s]

Downloading (…)l-00005-of-00005.bin:  96%|█████████▋| 4.01G/4.16G [01:05<00:02, 73.2MB/s]

Downloading (…)l-00005-of-00005.bin:  97%|█████████▋| 4.02G/4.16G [01:06<00:02, 63.0MB/s]

Downloading (…)l-00005-of-00005.bin:  97%|█████████▋| 4.04G/4.16G [01:06<00:01, 71.2MB/s]

Downloading (…)l-00005-of-00005.bin:  98%|█████████▊| 4.06G/4.16G [01:06<00:01, 79.2MB/s]

Downloading (…)l-00005-of-00005.bin:  98%|█████████▊| 4.07G/4.16G [01:06<00:01, 67.7MB/s]

Downloading (…)l-00005-of-00005.bin:  98%|█████████▊| 4.08G/4.16G [01:07<00:01, 63.7MB/s]

Downloading (…)l-00005-of-00005.bin:  98%|█████████▊| 4.09G/4.16G [01:07<00:01, 54.7MB/s]

Downloading (…)l-00005-of-00005.bin:  99%|█████████▊| 4.10G/4.16G [01:07<00:01, 46.8MB/s]

Downloading (…)l-00005-of-00005.bin:  99%|█████████▉| 4.12G/4.16G [01:07<00:00, 62.7MB/s]

Downloading (…)l-00005-of-00005.bin:  99%|█████████▉| 4.13G/4.16G [01:07<00:00, 64.8MB/s]

Downloading (…)l-00005-of-00005.bin: 100%|█████████▉| 4.15G/4.16G [01:08<00:00, 79.3MB/s]
Downloading (…)l-00005-of-00005.bin: 100%|██████████| 4.16G/4.16G [01:08<00:00, 61.0MB/s]

Downloading shards: 100%|██████████| 5/5 [06:31<00:00, 76.36s/it]
Downloading shards: 100%|██████████| 5/5 [06:31<00:00, 78.36s/it]

Loading checkpoint shards:   0%|          | 0/5 [00:00<?, ?it/s]
Loading checkpoint shards:  20%|██        | 1/5 [00:14<00:58, 14.53s/it]
Loading checkpoint shards:  40%|████      | 2/5 [00:29<00:44, 14.77s/it]
Loading checkpoint shards:  60%|██████    | 3/5 [00:44<00:29, 14.85s/it]
Loading checkpoint shards:  80%|████████  | 4/5 [01:00<00:15, 15.17s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:12<00:00, 14.21s/it]
Loading checkpoint shards: 100%|██████████| 5/5 [01:12<00:00, 14.52s/it]

Downloading (…)neration_config.json:   0%|          | 0.00/116 [00:00<?, ?B/s]
Downloading (…)neration_config.json: 100%|██████████| 116/116 [00:00<00:00, 86.6kB/s]
Get OpenAssistant/reward-model-deberta-v3-large-v2 model

Downloading (…)okenizer_config.json:   0%|          | 0.00/455 [00:00<?, ?B/s]
Downloading (…)okenizer_config.json: 100%|██████████| 455/455 [00:00<00:00, 323kB/s]

Downloading spm.model:   0%|          | 0.00/2.46M [00:00<?, ?B/s]
Downloading spm.model: 100%|██████████| 2.46M/2.46M [00:00<00:00, 65.2MB/s]

Downloading (…)/main/tokenizer.json:   0%|          | 0.00/8.66M [00:00<?, ?B/s]
Downloading (…)/main/tokenizer.json: 100%|██████████| 8.66M/8.66M [00:00<00:00, 378MB/s]

Downloading (…)in/added_tokens.json:   0%|          | 0.00/23.0 [00:00<?, ?B/s]
Downloading (…)in/added_tokens.json: 100%|██████████| 23.0/23.0 [00:00<00:00, 17.1kB/s]

Downloading (…)cial_tokens_map.json:   0%|          | 0.00/173 [00:00<?, ?B/s]
Downloading (…)cial_tokens_map.json: 100%|██████████| 173/173 [00:00<00:00, 128kB/s]

Downloading (…)lve/main/config.json:   0%|          | 0.00/993 [00:00<?, ?B/s]
Downloading (…)lve/main/config.json: 100%|██████████| 993/993 [00:00<00:00, 644kB/s]
device_map: {'': 0}

Downloading pytorch_model.bin:   0%|          | 0.00/1.74G [00:00<?, ?B/s]
Downloading pytorch_model.bin:   2%|▏         | 31.5M/1.74G [00:00<00:15, 111MB/s]
Downloading pytorch_model.bin:   3%|▎         | 52.4M/1.74G [00:00<00:15, 107MB/s]
Downloading pytorch_model.bin:   4%|▍         | 73.4M/1.74G [00:00<00:20, 79.8MB/s]
Downloading pytorch_model.bin:   5%|▍         | 83.9M/1.74G [00:01<00:25, 64.7MB/s]
Downloading pytorch_model.bin:   6%|▌         | 105M/1.74G [00:01<00:24, 67.1MB/s] 
Downloading pytorch_model.bin:   7%|▋         | 126M/1.74G [00:01<00:18, 88.1MB/s]
Downloading pytorch_model.bin:   8%|▊         | 147M/1.74G [00:02<00:24, 64.5MB/s]
Downloading pytorch_model.bin:   9%|▉         | 157M/1.74G [00:02<00:26, 59.9MB/s]
Downloading pytorch_model.bin:  11%|█         | 189M/1.74G [00:02<00:22, 68.9MB/s]
Downloading pytorch_model.bin:  11%|█▏        | 199M/1.74G [00:02<00:21, 73.0MB/s]
Downloading pytorch_model.bin:  12%|█▏        | 210M/1.74G [00:02<00:23, 65.5MB/s]
Downloading pytorch_model.bin:  13%|█▎        | 231M/1.74G [00:03<00:17, 87.1MB/s]
Downloading pytorch_model.bin:  14%|█▍        | 252M/1.74G [00:03<00:17, 82.9MB/s]
Downloading pytorch_model.bin:  15%|█▌        | 262M/1.74G [00:03<00:21, 67.4MB/s]
Downloading pytorch_model.bin:  16%|█▋        | 283M/1.74G [00:03<00:18, 77.6MB/s]
Downloading pytorch_model.bin:  17%|█▋        | 294M/1.74G [00:03<00:20, 70.0MB/s]
Downloading pytorch_model.bin:  18%|█▊        | 315M/1.74G [00:04<00:20, 70.8MB/s]
Downloading pytorch_model.bin:  19%|█▊        | 325M/1.74G [00:04<00:19, 72.9MB/s]
Downloading pytorch_model.bin:  20%|█▉        | 346M/1.74G [00:04<00:19, 71.1MB/s]
Downloading pytorch_model.bin:  21%|██        | 367M/1.74G [00:05<00:19, 70.8MB/s]
Downloading pytorch_model.bin:  23%|██▎       | 398M/1.74G [00:05<00:16, 79.0MB/s]
Downloading pytorch_model.bin:  24%|██▍       | 419M/1.74G [00:05<00:16, 81.8MB/s]
Downloading pytorch_model.bin:  26%|██▌       | 451M/1.74G [00:05<00:14, 89.8MB/s]
Downloading pytorch_model.bin:  27%|██▋       | 472M/1.74G [00:06<00:16, 77.2MB/s]
Downloading pytorch_model.bin:  29%|██▉       | 503M/1.74G [00:06<00:14, 84.9MB/s]
Downloading pytorch_model.bin:  30%|███       | 524M/1.74G [00:06<00:15, 77.1MB/s]
Downloading pytorch_model.bin:  32%|███▏      | 556M/1.74G [00:07<00:15, 76.6MB/s]
Downloading pytorch_model.bin:  33%|███▎      | 566M/1.74G [00:07<00:15, 74.0MB/s]
Downloading pytorch_model.bin:  33%|███▎      | 577M/1.74G [00:07<00:19, 59.9MB/s]
Downloading pytorch_model.bin:  34%|███▎      | 587M/1.74G [00:08<00:19, 59.1MB/s]
Downloading pytorch_model.bin:  34%|███▍      | 598M/1.74G [00:08<00:19, 59.1MB/s]
Downloading pytorch_model.bin:  35%|███▍      | 608M/1.74G [00:08<00:20, 56.0MB/s]
Downloading pytorch_model.bin:  36%|███▌      | 629M/1.74G [00:08<00:17, 63.6MB/s]
Downloading pytorch_model.bin:  37%|███▋      | 650M/1.74G [00:08<00:13, 83.3MB/s]
Downloading pytorch_model.bin:  38%|███▊      | 661M/1.74G [00:09<00:16, 64.2MB/s]
Downloading pytorch_model.bin:  39%|███▉      | 682M/1.74G [00:09<00:15, 67.5MB/s]
Downloading pytorch_model.bin:  41%|████      | 713M/1.74G [00:09<00:13, 76.4MB/s]
Downloading pytorch_model.bin:  42%|████▏     | 734M/1.74G [00:10<00:14, 69.3MB/s]
Downloading pytorch_model.bin:  43%|████▎     | 744M/1.74G [00:10<00:14, 71.0MB/s]
Downloading pytorch_model.bin:  44%|████▍     | 765M/1.74G [00:10<00:14, 68.2MB/s]
Downloading pytorch_model.bin:  45%|████▌     | 786M/1.74G [00:10<00:15, 63.3MB/s]
Downloading pytorch_model.bin:  46%|████▌     | 797M/1.74G [00:11<00:13, 68.1MB/s]
Downloading pytorch_model.bin:  47%|████▋     | 818M/1.74G [00:11<00:14, 64.6MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 828M/1.74G [00:11<00:13, 66.0MB/s]
Downloading pytorch_model.bin:  48%|████▊     | 839M/1.74G [00:11<00:15, 59.8MB/s]
Downloading pytorch_model.bin:  50%|█████     | 870M/1.74G [00:12<00:13, 65.6MB/s]
Downloading pytorch_model.bin:  51%|█████     | 881M/1.74G [00:12<00:13, 64.3MB/s]
Downloading pytorch_model.bin:  51%|█████     | 891M/1.74G [00:12<00:17, 49.4MB/s]
Downloading pytorch_model.bin:  52%|█████▏    | 902M/1.74G [00:12<00:15, 53.7MB/s]
Downloading pytorch_model.bin:  53%|█████▎    | 923M/1.74G [00:13<00:13, 60.3MB/s]
Downloading pytorch_model.bin:  54%|█████▍    | 944M/1.74G [00:13<00:12, 64.0MB/s]
Downloading pytorch_model.bin:  56%|█████▌    | 975M/1.74G [00:13<00:09, 76.7MB/s]
Downloading pytorch_model.bin:  57%|█████▋    | 986M/1.74G [00:13<00:10, 74.2MB/s]
Downloading pytorch_model.bin:  57%|█████▋    | 996M/1.74G [00:14<00:12, 62.0MB/s]
Downloading pytorch_model.bin:  58%|█████▊    | 1.02G/1.74G [00:14<00:08, 82.3MB/s]
Downloading pytorch_model.bin:  59%|█████▉    | 1.03G/1.74G [00:14<00:09, 76.9MB/s]
Downloading pytorch_model.bin:  60%|██████    | 1.05G/1.74G [00:14<00:09, 70.0MB/s]
Downloading pytorch_model.bin:  61%|██████▏   | 1.07G/1.74G [00:14<00:07, 88.3MB/s]
Downloading pytorch_model.bin:  63%|██████▎   | 1.09G/1.74G [00:15<00:07, 83.7MB/s]
Downloading pytorch_model.bin:  63%|██████▎   | 1.10G/1.74G [00:15<00:09, 67.2MB/s]
Downloading pytorch_model.bin:  64%|██████▍   | 1.11G/1.74G [00:15<00:09, 64.9MB/s]
Downloading pytorch_model.bin:  65%|██████▌   | 1.13G/1.74G [00:16<00:09, 61.4MB/s]
Downloading pytorch_model.bin:  66%|██████▋   | 1.15G/1.74G [00:16<00:10, 58.5MB/s]
Downloading pytorch_model.bin:  67%|██████▋   | 1.17G/1.74G [00:16<00:07, 76.7MB/s]
Downloading pytorch_model.bin:  69%|██████▊   | 1.20G/1.74G [00:16<00:06, 80.0MB/s]
Downloading pytorch_model.bin:  69%|██████▉   | 1.21G/1.74G [00:17<00:07, 70.5MB/s]
Downloading pytorch_model.bin:  70%|██████▉   | 1.22G/1.74G [00:17<00:07, 69.3MB/s]
Downloading pytorch_model.bin:  71%|███████   | 1.24G/1.74G [00:17<00:07, 63.4MB/s]
Downloading pytorch_model.bin:  72%|███████▏  | 1.25G/1.74G [00:17<00:07, 69.1MB/s]
Downloading pytorch_model.bin:  72%|███████▏  | 1.26G/1.74G [00:17<00:07, 64.9MB/s]
Downloading pytorch_model.bin:  73%|███████▎  | 1.27G/1.74G [00:18<00:07, 64.0MB/s]
Downloading pytorch_model.bin:  74%|███████▍  | 1.29G/1.74G [00:18<00:06, 69.2MB/s]
Downloading pytorch_model.bin:  75%|███████▌  | 1.31G/1.74G [00:18<00:05, 82.4MB/s]
Downloading pytorch_model.bin:  76%|███████▌  | 1.32G/1.74G [00:18<00:05, 73.2MB/s]
Downloading pytorch_model.bin:  77%|███████▋  | 1.33G/1.74G [00:18<00:06, 67.4MB/s]
Downloading pytorch_model.bin:  77%|███████▋  | 1.34G/1.74G [00:19<00:08, 46.2MB/s]
Downloading pytorch_model.bin:  78%|███████▊  | 1.36G/1.74G [00:19<00:07, 47.9MB/s]
Downloading pytorch_model.bin:  80%|███████▉  | 1.38G/1.74G [00:19<00:05, 64.3MB/s]
Downloading pytorch_model.bin:  80%|████████  | 1.39G/1.74G [00:20<00:07, 45.6MB/s]
Downloading pytorch_model.bin:  81%|████████▏ | 1.42G/1.74G [00:20<00:05, 54.9MB/s]
Downloading pytorch_model.bin:  83%|████████▎ | 1.44G/1.74G [00:20<00:04, 73.2MB/s]
Downloading pytorch_model.bin:  84%|████████▎ | 1.46G/1.74G [00:20<00:03, 79.0MB/s]
Downloading pytorch_model.bin:  84%|████████▍ | 1.47G/1.74G [00:21<00:03, 69.5MB/s]
Downloading pytorch_model.bin:  86%|████████▌ | 1.50G/1.74G [00:21<00:03, 75.7MB/s]
Downloading pytorch_model.bin:  87%|████████▋ | 1.52G/1.74G [00:21<00:03, 65.9MB/s]
Downloading pytorch_model.bin:  88%|████████▊ | 1.53G/1.74G [00:22<00:02, 70.4MB/s]
Downloading pytorch_model.bin:  89%|████████▉ | 1.55G/1.74G [00:22<00:02, 76.8MB/s]
Downloading pytorch_model.bin:  90%|█████████ | 1.57G/1.74G [00:22<00:02, 77.4MB/s]
Downloading pytorch_model.bin:  92%|█████████▏| 1.59G/1.74G [00:22<00:01, 92.4MB/s]
Downloading pytorch_model.bin:  93%|█████████▎| 1.61G/1.74G [00:23<00:02, 60.2MB/s]
Downloading pytorch_model.bin:  93%|█████████▎| 1.63G/1.74G [00:23<00:02, 51.5MB/s]
Downloading pytorch_model.bin:  94%|█████████▍| 1.64G/1.74G [00:23<00:01, 55.0MB/s]
Downloading pytorch_model.bin:  95%|█████████▌| 1.66G/1.74G [00:24<00:01, 55.7MB/s]
Downloading pytorch_model.bin:  96%|█████████▌| 1.67G/1.74G [00:24<00:01, 55.7MB/s]
Downloading pytorch_model.bin:  96%|█████████▋| 1.68G/1.74G [00:24<00:01, 48.9MB/s]
Downloading pytorch_model.bin:  98%|█████████▊| 1.70G/1.74G [00:24<00:00, 61.1MB/s]
Downloading pytorch_model.bin:  98%|█████████▊| 1.71G/1.74G [00:25<00:00, 42.9MB/s]
Downloading pytorch_model.bin:  99%|█████████▉| 1.73G/1.74G [00:25<00:00, 46.2MB/s]
Downloading pytorch_model.bin: 100%|██████████| 1.74G/1.74G [00:25<00:00, 51.9MB/s]
Downloading pytorch_model.bin: 100%|██████████| 1.74G/1.74G [00:25<00:00, 67.2MB/s]
/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/deprecation.py:43: UserWarning: You have unused kwarg parameters in Row, please remove them: {'scale': 1}
  warnings.warn(
Running on local URL:  http://0.0.0.0:7860

To create a public link, set `share=True` in `launch()`.
Started GUI
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Task exception was never retrieved
future: <Task finished name='xsce894h9ta_5' coro=<Queue.process_events() done, defined at /home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/queueing.py:343> exception=WebSocketDisconnect(1001)>
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/queueing.py", line 347, in process_events
    client_awake = await self.gather_event_data(event)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/queueing.py", line 220, in gather_event_data
    data, client_awake = await self.get_message(event, timeout=receive_timeout)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/queueing.py", line 453, in get_message
    data = await asyncio.wait_for(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/asyncio/tasks.py", line 494, in wait_for
    return fut.result()
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/starlette/websockets.py", line 133, in receive_json
    self._raise_on_disconnect(message)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/starlette/websockets.py", line 105, in _raise_on_disconnect
    raise WebSocketDisconnect(message["code"])
starlette.websockets.WebSocketDisconnect: 1001
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
WARNING: Special characters in prompt
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "app.py", line 836, in bot
    for output in fun1(*tuple(args_list)):
  File "app.py", line 1247, in evaluate
    for output in CallbackToGenerator(generate, callback=None, **gen_kwargs):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/_collections_abc.py", line 317, in __next__
    return self.send(None)
  File "/home/user/app/stopping.py", line 119, in send
    return self._put('send', value)
  File "/home/user/app/stopping.py", line 111, in _put
    raise val
  File "/home/user/app/stopping.py", line 95, in thread_func
    ret = func(callback=val_callback, **self.kwargs)
  File "app.py", line 1245, in generate
    model.generate(**kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 1437, in generate
    return self.greedy_search(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 2248, in greedy_search
    outputs = self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 662, in forward
    outputs = self.gpt_neox(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 553, in forward
    outputs = layer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 320, in forward
    attention_layer_outputs = self.attention(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 152, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 226, in _attn
    attn_weights = attn_weights.to(value.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 14.00 MiB (GPU 0; 22.20 GiB total capacity; 20.53 GiB already allocated; 12.12 MiB free; 21.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "app.py", line 836, in bot
    for output in fun1(*tuple(args_list)):
  File "app.py", line 1247, in evaluate
    for output in CallbackToGenerator(generate, callback=None, **gen_kwargs):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/_collections_abc.py", line 317, in __next__
    return self.send(None)
  File "/home/user/app/stopping.py", line 119, in send
    return self._put('send', value)
  File "/home/user/app/stopping.py", line 111, in _put
    raise val
  File "/home/user/app/stopping.py", line 95, in thread_func
    ret = func(callback=val_callback, **self.kwargs)
  File "app.py", line 1245, in generate
    model.generate(**kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 1437, in generate
    return self.greedy_search(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 2248, in greedy_search
    outputs = self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 676, in forward
    lm_logits = self.embed_out(hidden_states)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/linear.py", line 114, in forward
    return F.linear(input, self.weight, self.bias)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.20 GiB total capacity; 20.71 GiB already allocated; 6.12 MiB free; 21.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "app.py", line 836, in bot
    for output in fun1(*tuple(args_list)):
  File "app.py", line 1247, in evaluate
    for output in CallbackToGenerator(generate, callback=None, **gen_kwargs):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/_collections_abc.py", line 317, in __next__
    return self.send(None)
  File "/home/user/app/stopping.py", line 119, in send
    return self._put('send', value)
  File "/home/user/app/stopping.py", line 111, in _put
    raise val
  File "/home/user/app/stopping.py", line 95, in thread_func
    ret = func(callback=val_callback, **self.kwargs)
  File "app.py", line 1245, in generate
    model.generate(**kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 1437, in generate
    return self.greedy_search(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 2248, in greedy_search
    outputs = self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 662, in forward
    outputs = self.gpt_neox(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 553, in forward
    outputs = layer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 335, in forward
    mlp_output = self.mlp(self.post_attention_layernorm(hidden_states))
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 295, in forward
    hidden_states = self.dense_h_to_4h(hidden_states)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/nn/modules.py", line 320, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 500, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
    out32, Sout32 = F.igemmlt(C32A, state.CxB, SA, state.SB)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/functional.py", line 1367, in igemmlt
    out, Sout = get_transform_buffer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/functional.py", line 363, in get_transform_buffer
    return init_func((rows, cols), dtype=dtype, device=device), state
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.20 GiB total capacity; 20.72 GiB already allocated; 6.12 MiB free; 21.15 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "app.py", line 836, in bot
    for output in fun1(*tuple(args_list)):
  File "app.py", line 1247, in evaluate
    for output in CallbackToGenerator(generate, callback=None, **gen_kwargs):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/_collections_abc.py", line 317, in __next__
    return self.send(None)
  File "/home/user/app/stopping.py", line 119, in send
    return self._put('send', value)
  File "/home/user/app/stopping.py", line 111, in _put
    raise val
  File "/home/user/app/stopping.py", line 95, in thread_func
    ret = func(callback=val_callback, **self.kwargs)
  File "app.py", line 1245, in generate
    model.generate(**kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 1437, in generate
    return self.greedy_search(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 2248, in greedy_search
    outputs = self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 662, in forward
    outputs = self.gpt_neox(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 553, in forward
    outputs = layer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 320, in forward
    attention_layer_outputs = self.attention(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 152, in forward
    attn_output, attn_weights = self._attn(query, key, value, attention_mask, head_mask)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 206, in _attn
    attn_scores = torch.baddbmm(
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 28.00 MiB (GPU 0; 22.20 GiB total capacity; 20.69 GiB already allocated; 6.12 MiB free; 21.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF
Using pad_token, but it is not set yet.
Traceback (most recent call last):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/routes.py", line 401, in run_predict
    output = await app.get_blocks().process_api(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1302, in process_api
    result = await self.call_function(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1039, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/utils.py", line 491, in async_iteration
    return next(iterator)
  File "app.py", line 836, in bot
    for output in fun1(*tuple(args_list)):
  File "app.py", line 1247, in evaluate
    for output in CallbackToGenerator(generate, callback=None, **gen_kwargs):
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/_collections_abc.py", line 317, in __next__
    return self.send(None)
  File "/home/user/app/stopping.py", line 119, in send
    return self._put('send', value)
  File "/home/user/app/stopping.py", line 111, in _put
    raise val
  File "/home/user/app/stopping.py", line 95, in thread_func
    ret = func(callback=val_callback, **self.kwargs)
  File "app.py", line 1245, in generate
    model.generate(**kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/utils/_contextlib.py", line 115, in decorate_context
    return func(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 1437, in generate
    return self.greedy_search(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/generation/utils.py", line 2248, in greedy_search
    outputs = self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 662, in forward
    outputs = self.gpt_neox(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 553, in forward
    outputs = layer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 335, in forward
    mlp_output = self.mlp(self.post_attention_layernorm(hidden_states))
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/gpt_neox/modeling_gpt_neox.py", line 295, in forward
    hidden_states = self.dense_h_to_4h(hidden_states)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/nn/modules.py", line 320, in forward
    out = bnb.matmul(x, self.weight, bias=self.bias, state=self.state)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 500, in matmul
    return MatMul8bitLt.apply(A, B, out, bias, state)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/autograd/function.py", line 506, in apply
    return super().apply(*args, **kwargs)  # type: ignore[misc]
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/autograd/_functions.py", line 397, in forward
    out32, Sout32 = F.igemmlt(C32A, state.CxB, SA, state.SB)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/functional.py", line 1367, in igemmlt
    out, Sout = get_transform_buffer(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/bitsandbytes/functional.py", line 363, in get_transform_buffer
    return init_func((rows, cols), dtype=dtype, device=device), state
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 2.00 MiB (GPU 0; 22.20 GiB total capacity; 20.70 GiB already allocated; 4.12 MiB free; 21.14 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

@pseudotensor
Copy link
Collaborator Author 

 File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/gradio/blocks.py", line 1025, in call_function
    prediction = await anyio.to_thread.run_sync(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/to_thread.py", line 31, in run_sync
    return await get_asynclib().run_sync_in_worker_thread(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 937, in run_sync_in_worker_thread
    return await future
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/anyio/_backends/_asyncio.py", line 867, in run
    result = context.run(func, *args)
  File "app.py", line 764, in score_last_response
    score = torch.sigmoid(smodel(**inputs).logits[0]).cpu().detach().numpy()[0]
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1311, in forward
    outputs = self.deberta(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 1083, in forward
    encoder_outputs = self.encoder(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 521, in forward
    output_states = layer_module(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 362, in forward
    attention_output = self.attention(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 293, in forward
    self_output = self.self(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/torch/nn/modules/module.py", line 1501, in _call_impl
    return forward_call(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/accelerate/hooks.py", line 165, in new_forward
    output = old_forward(*args, **kwargs)
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 728, in forward
    rel_att = self.disentangled_attention_bias(
  File "/home/user/.pyenv/versions/3.8.9/lib/python3.8/site-packages/transformers/models/deberta_v2/modeling_deberta_v2.py", line 811, in disentangled_attention_bias
    score += c2p_att / scale.to(dtype=c2p_att.dtype)
torch.cuda.OutOfMemoryError: CUDA out of memory. Tried to allocate 20.00 MiB (GPU 0; 22.20 GiB total capacity; 20.85 GiB already allocated; 14.12 MiB free; 21.12 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

pseudotensor added a commit that referenced this issue Apr 20, 2023
@pseudotensor
Merge pull request #55 from h2oai/fixes48
0915f72 
Fixes #48
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@pseudotensor