RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

[bonillst]

Got the following error when running quicktest.sh

RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

The full output:

++ cryodrgn train_vae data/hand.mrcs -o output/toy_recon_vae --lr .0001 --seed 0 --poses data/hand_rot.pkl --zdim 10
2021-10-20 22:13:46 /data/home/bonilla/.conda/envs/cryodrgn_ver2/bin/cryodrgn train_vae data/hand.mrcs -o output/toy_recon_vae --lr .0001 --seed 0 --poses data/hand_rot.pkl --zdim 10
2021-10-20 22:13:47 Namespace(activation='relu', amp=False, batch_size=8, beta=None, beta_control=None, checkpoint=1, ctf=None, datadir=None, do_pose_sgd=False, domain='fourier', emb_type='quat', enc_mask=None, encode_mode='resid', func=<function main at 0x7fa40b7f6f80>, ind=None, invert_data=True, lazy=False, load=None, log_interval=1000, lr=0.0001, multigpu=False, norm=None, num_epochs=20, outdir='/data/home/bonilla/cryodrgn/testing/output/toy_recon_vae', particles='/data/home/bonilla/cryodrgn/testing/data/hand.mrcs', pdim=256, pe_dim=None, pe_type='geom_lowf', players=3, pose_lr=0.0003, poses='/data/home/bonilla/cryodrgn/testing/data/hand_rot.pkl', pretrain=1, qdim=256, qlayers=3, relion31=False, seed=0, tilt=None, tilt_deg=45, use_real=False, verbose=False, wd=0, window=True, zdim=10)
2021-10-20 22:13:47 Use cuda True
2021-10-20 22:13:47 Loaded 100 64x64 images
2021-10-20 22:13:47 Normalized HT by 0 +/- 94.426513671875
2021-10-20 22:13:47 WARNING: No translations provided
2021-10-20 22:13:49 Using circular lattice with radius 32
2021-10-20 22:13:49 HetOnlyVAE(
(encoder): ResidLinearMLP(
(main): Sequential(
(0): Linear(in_features=3208, out_features=256, bias=True)
(1): ReLU()
(2): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(3): ReLU()
(4): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(5): ReLU()
(6): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(7): ReLU()
(8): Linear(in_features=256, out_features=20, bias=True)
)
)
(decoder): FTPositionalDecoder(
(decoder): ResidLinearMLP(
(main): Sequential(
(0): Linear(in_features=202, out_features=256, bias=True)
(1): ReLU()
(2): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(3): ReLU()
(4): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(5): ReLU()
(6): ResidLinear(
(linear): Linear(in_features=256, out_features=256, bias=True)
)
(7): ReLU()
(8): Linear(in_features=256, out_features=2, bias=True)
)
)
)
)
2021-10-20 22:13:49 1273878 parameters in model
2021-10-20 22:13:49 1024020 parameters in encoder
2021-10-20 22:13:49 249858 parameters in decoder
Traceback (most recent call last):
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/bin/cryodrgn", line 33, in
sys.exit(load_entry_point('cryodrgn==0.3.2b0', 'console_scripts', 'cryodrgn')())
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/cryodrgn-0.3.2b0-py3.7.egg/cryodrgn/main.py", line 52, in main
args.func(args)
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/cryodrgn-0.3.2b0-py3.7.egg/cryodrgn/commands/train_vae.py", line 411, in main
for minibatch in data_generator:
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 521, in next
data = self._next_data()
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 560, in _next_data
index = self._next_index() # may raise StopIteration
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/torch/utils/data/dataloader.py", line 512, in _next_index
return next(self._sampler_iter) # may raise StopIteration
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 229, in iter
for idx in self.sampler:
File "/data/home/bonilla/.conda/envs/cryodrgn_ver2/lib/python3.7/site-packages/torch/utils/data/sampler.py", line 126, in iter
yield from torch.randperm(n, generator=generator).tolist()
RuntimeError: Expected a 'cuda' device type for generator but found 'cpu'

[zhonge]

What version of pytorch are you using? This might be a compatibility issue with pytorch 1.9

$ python
>> import torch
>> torch.__version__

[bonillst]

Hi, the version of pytorch is 1.10.0

print(torch.version)
1.10.0

[loganjsweezy]

I am also experiencing the same issue, also with pytorch version 1.10.0. Has anyone found a workaround for this issue?

Let me know if I can provide any information about my build that may assist

[zhonge]

This is a compatibility issue with the latest versions of pytorch (1.9+). I will address this in a future version of cryodrgn.

For now, you can install an earlier version of pytorch and it should work fine. See
#66

[bonillst]

Sorry, I missed the note about the incompatibility with pytorch 1.9+ in the installation instructions. Yes, it worked once I installed an older version of pytorch. Thanks for your help!

[zhonge]

This incompatibility with the latest version of pytorch is fixed in the top-of-tree and will be available in the upcoming version 1.0.0 release.