[Question] Is DQN actually Double DQN?

[oliverc1623]

❓ Question

Hi, first off great work on making Stable-Baselines3 an excellent resource for deep reinforcement learning practitioners.

I noticed that your DQN implementation features a target q network which resembles Google's Deep Mind paper, Deep Reinforcement Learning with Double Q-learning. Meanwhile, Neural Fitted Q Iteration, by Riedmiller, calculates the target using the "current estimate" of the Q function. I am looking for clarification if DQN is truly a Double DQN. I hope to use this information to accurately hopefully implement prioritized experience replay based off of your DQN implementation.

Thanks,
Oliver

Checklist

• I have checked that there is no similar issue in the repo
• I have read the documentation
• If code there is, it is minimal and working
• If code there is, it is formatted using the markdown code blocks for both code and stack traces.

[araffin]

Duplicate of #487

It is not, current DQN is the vanilla DQN (cf. doc), but we are working (and welcome help) on #622 (there is a PR for PER).
However, we do provide QR-DQN in SB3 contrib repo.

[oliverc1623]

Thanks for the clarification, will read up on #487 and the papers more throughly.