MPDQN throw a RuntimeError

[youhu868]

from ding.model.template.pdqn import *
obs_dim, seq_len, bs = 128, 64, 32
action_mask = [[0,0,0,0],[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]
action_space = {"action_type_shape": torch.LongTensor([5]), "action_args_shape": torch.LongTensor([4])}

pdqn_model = PDQN(obs_dim, EasyDict(action_space), multi_pass=True, action_mask=action_mask)

obs = torch.rand(seq_len, bs, obs_dim)
action_args = pdqn_model.forward(obs, "compute_continuous")
action_type = pdqn_model.forward({"state": obs, "action_args": action_args["action_args"]}, "compute_discrete")

==================================================================================================
Traceback (most recent call last):
File "/home/xxx/anaconda3/envs/vnpy/lib/python3.7/site-packages/IPython/core/interactiveshell.py", line 3319, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
action_type = pdqn_model.forward({"state": obs, "action_args": action_args["action_args"]}, "compute_discrete")
File "/home/xxx/anaconda3/envs/vnpy/lib/python3.7/site-packages/ding/model/template/pdqn.py", line 137, in forward
return getattr(self, mode)(inputs)
File "/home/xxx/anaconda3/envs/vnpy/lib/python3.7/site-packages/ding/model/template/pdqn.py", line 182, in compute_discrete
mp_action.scatter_(dim=-1, index=index, src=action_args.unsqueeze(-1))
RuntimeError: Index tensor must have the same number of dimensions as src tensor

[PaParaZz1]

This problem is because that PDQN doesn't support 3-dimension obs like (T, B, N), however, the operations in T and B are usually independent and parallel, we can view them as a new batch=T*B dimension, for example:

from ding.model.template.pdqn import *
obs_dim, seq_len, bs = 128, 64, 32
action_mask = [[0,0,0,0],[1,0,0,0],[0,1,0,0],[0,0,1,0],[0,0,0,1]]
action_space = {"action_type_shape": torch.LongTensor([5]), "action_args_shape": torch.LongTensor([4])}

pdqn_model = PDQN(obs_dim, EasyDict(action_space), multi_pass=True, action_mask=action_mask)

obs = torch.rand(seq_len, bs, obs_dim)
obs = obs.view(-1, obs_dim)
action_args = pdqn_model.forward(obs, "compute_continuous")
action_type = pdqn_model.forward({"state": obs, "action_args": action_args["action_args"]}, "compute_discrete")
action_args["action_args"] = action_args["action_args"].view(seq_len, bs, -1)
action_type["logit"] = action_type["logit"].view(seq_len, bs, -1)

[youhu868]

ok, i see, thanks