You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I ran into an OOM error while trying to train the model, on a 1-GPU workstation (GTX 1080TI). A snip of the error log can be seen below. Has anyone been successful in training this model with similar hardware?
Thanks in advance!
ResourceExhaustedError (see above for traceback): OOM when allocating tensor with shape[512,98,200] and type float on /job:localhost/replica:0/task:0/device:GPU:0 by allocator GPU_0_bfc
[[Node: block-1/align/sub_1/mul_5 = Mul[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:GPU:0"](block-1/align/sub_1/mul_2, block-1/align/sub_1/add_1)]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
[[Node: stack_8/_159 = _Recv[client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_name="edge_6954_stack_8", tensor_type=DT_FLOAT, _device="/job:localhost/replica:0/task:0/device:CPU:0"]()]]
Hint: If you want to see a list of allocated tensors when OOM happens, add report_tensor_allocations_upon_oom to RunOptions for current allocation info.
Edit: Decreasing batch size to 64 (default was 512) fixed this issue. I will try to see what's the maximum batch size that the 1080TI can handle.
The text was updated successfully, but these errors were encountered:
Hi all,
I ran into an OOM error while trying to train the model, on a 1-GPU workstation (GTX 1080TI). A snip of the error log can be seen below. Has anyone been successful in training this model with similar hardware?
Thanks in advance!
Edit: Decreasing batch size to 64 (default was 512) fixed this issue. I will try to see what's the maximum batch size that the 1080TI can handle.
The text was updated successfully, but these errors were encountered: