[BUG] IndexError / Runtime Error with torch.nn.TransformerEncoder
#1795
Assignees
Labels
bug
Something isn't working
Comments
|
I was able to get your model running. You need to make your dataset type import os
import torch
import torch.nn as nn
from torch.utils.data import DataLoader, Dataset
import deepspeed
class BoringDataset(Dataset):
def __init__(self) -> None:
super().__init__()
self.data = torch.randn(200, 128, 1024).half()
def __len__(self):
return len(self.data)
def __getitem__(self, index):
return self.data[index, :, :], self.data[index, :, :]
train_set = BoringDataset()
train_loader = DataLoader(train_set, batch_size=32)
ds_config = {
"train_micro_batch_size_per_gpu": 2,
"gradient_accumulation_steps": 1,
"steps_per_print": 1,
"zero_optimization": {
"stage": 3,
"stage3_max_live_parameters": 1e9,
"stage3_max_reuse_distance": 1e9,
"stage3_param_persitence_threshold": 1e5,
"stage3_prefetch_bucket_size": 5e7,
"contiguous_gradients": True,
"overlap_comm": True,
"reduce_bucket_size": 90000000,
"sub_group_size": 1e8,
"offload_param": {"device": "cpu", "pin_memory": True},
"offload_optimizer": {"device": "cpu", "pin_memory": True},
},
"optimizer": {"type": "Adam", "params": {"lr": 0.001, "betas": [0.9, 0.95]}},
"fp16": {
"enabled": True,
"loss_scale": 0,
"loss_scale_window": 1000,
"hysteresis": 2,
"min_loss_scale": 1,
},
"gradient_clipping": 1.0,
"wall_clock_breakdown": True,
"zero_allow_untested_optimizer": False,
"aio": {
"block_size": 1048576,
"queue_depth": 16,
"single_submit": False,
"overlap_events": True,
"thread_count": 2,
},
}
encoder_layer = nn.TransformerEncoderLayer(d_model=1024, nhead=8)
transformer = nn.TransformerEncoder(encoder_layer=encoder_layer, num_layers=10)
model, _, _, _ = deepspeed.initialize(
config=ds_config, model=transformer, model_parameters=transformer.parameters()
)
loss_fn = nn.MSELoss()
model.train()
rank = int(os.getenv("RANK", "0"))
for step, (batch, label) in enumerate(train_loader):
if rank == 0:
print("step:", step)
batch = batch.cuda(rank)
label = label.cuda(rank)
output = model(batch)
model.backward(loss_fn(output, label))
model.step() |
|
Closing due to inactivity, please reopen if you are unable to run the model with the updated script. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the bug
A clear and concise description of what the bug is.
To Reproduce
I'm fairly new to DeepSpeed, and I tried deepspeed zero-3 and offload with pytorch transformer.
And I write a simple test code:
But I encountered traceback
RuntimeError: expected scalar type Float but found Half
So, I refer to previous issues but find nothing helps. I guess I may half the input tensor manually. But I get another traceback
IndexError: list index out of range
sorry that I really can hardly find some hints in deepspeed docs or elsewhere, so I post a issue here
DeepSpeed JSON config
{ "train_micro_batch_size_per_gpu": 2 , "gradient_accumulation_steps": 1, "steps_per_print": 1, "zero_optimization": { "stage": 3, "stage3_max_live_parameters": 1e9, "stage3_max_reuse_distance": 1e9, "stage3_param_persitence_threshold": 1e5, "stage3_prefetch_bucket_size": 5e7, "contiguous_gradients": true, "overlap_comm": true, "reduce_bucket_size": 90000000, "sub_group_size": 1e8, "offload_param": { "device": "cpu", "pin_memory": true }, "offload_optimizer": { "device": "cpu", "pin_memory": true } }, "optimizer": { "type": "Adam", "params": { "lr": 0.001, "betas": [ 0.9, 0.95 ] } }, "fp16": { "enabled": true, "loss_scale": 0, "loss_scale_window": 1000, "hysteresis": 2, "min_loss_scale": 1 }, "gradient_clipping": 1.0, "wall_clock_breakdown": true, "zero_allow_untested_optimizer": false, "aio": { "block_size": 1048576, "queue_depth": 16, "single_submit": false, "overlap_events": true, "thread_count": 2 } }ds_report output
Please run
ds_reportto give us details about your setup.-------------------------------------------------- DeepSpeed C++/CUDA extension op report -------------------------------------------------- NOTE: Ops not installed will be just-in-time (JIT) compiled at runtime if needed. Op compatibility means that your system meet the required dependencies to JIT install the op. -------------------------------------------------- JIT compiled ops requires ninja ninja .................. [OKAY] -------------------------------------------------- op name ................ installed .. compatible -------------------------------------------------- cpu_adam ............... [YES] ...... [OKAY] cpu_adagrad ............ [YES] ...... [OKAY] fused_adam ............. [YES] ...... [OKAY] fused_lamb ............. [YES] ...... [OKAY] sparse_attn ............ [YES] ...... [OKAY] transformer ............ [YES] ...... [OKAY] stochastic_transformer . [YES] ...... [OKAY] async_io ............... [YES] ...... [OKAY] transformer_inference .. [YES] ...... [OKAY] utils .................. [YES] ...... [OKAY] quantizer .............. [YES] ...... [OKAY] -------------------------------------------------- DeepSpeed general environment info: torch install path ............... ['/chenjh02/miniconda3/envs/yuan/lib/python3.8/site-packages/torch'] torch version .................... 1.8.1+cu101 torch cuda version ............... 10.1 nvcc version ..................... 10.1 deepspeed install path ........... ['/chenjh02/miniconda3/envs/yuan/lib/python3.8/site-packages/deepspeed'] deepspeed info ................... 0.5.8, unknown, unknown deepspeed wheel compiled w. ...... torch 1.8, cuda 10.1Screenshots
If applicable, add screenshots to help explain your problem.
System info (please complete the following information):
Launcher context
Are you launching your experiment with the
deepspeedlauncher, MPI, or something else?deepspeed launcher, here's my script
Docker context
Are you using a specific docker image that you can share?
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: