-
Notifications
You must be signed in to change notification settings - Fork 141
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Doing Counterfactual Experience Replay #222
Comments
You can try overriding the MARLlib/marllib/marl/algos/core/CC/coma.py Lines 116 to 125 in 368c617
the signature for
As displayed in the You can manipulate the |
@Aequatio-Space Is this function called before or after both the policy gradient and value function gradient update in ippo or in between? |
@nikhil-pitta it is called before both the policy gradient and value function gradient. The pipeline is basically: extra_action_out_fn → postprocess_fn → loss_fn → compute_gradients → apply_gradients. |
@Aequatio-Space Thanks so much for the info! We would like to add counterfactual experience to DDPG before adding to the replay buffer or when pulling from the replay buffer to evaluate. Specifically, we want to augment our current step/collected experiences and add to the replay buffer for the algorithm to use. What callback function could we do to use this, and would it be the same for every offline policy? If not a callback, what would you suggest we do? |
@nikhil-pitta You mentioned "augment our current step/collected experiences and add to the replay buffer", and it sounds exactly what |
@Aequatio-Space Thank you for the response, and yes that function is exactly what I need! I tried using it for PPO by overriding the postprocess_fn for the IPPOTorchPolicy (https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/IL/ppo.py; line 31). I wanted to do the same thing for IQL, and saw this file https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/VD/iql_vdn_qmix.py, but I was unsure how to add a postprocess_fn or any other callback because it looks like a new policy is written up within this file that doesn't accept callback functions like the PPOTorchPolicy. Do you know what I can do to add callback functionality for IQL? |
@nikhil-pitta Note that |
@Aequatio-Space Thanks for the response again! So now I am trying use IQL, but an issue I'm facing is that I want each of my agents to have separate policies, but I run into the exception ValueError("joint Q learning does not support individual function"), which comes from setting share_policy="individual." This is not my desired behavior because I would like each agent to have their own policies, which is how I expected IQL to be implemented. Is there any way to get around this? |
I remember joint Q learning supports MARLlib/marllib/marl/algos/run_vd.py Lines 105 to 118 in 368c617
Try to adapt the code under this setting. |
I would like to try and do some counterfactual experience replay (specifically I want to use PPO and do CER on the critic). Is there something that exposes the replay buffer or something that allows me to add custom trajectories while training somehow?
Also, how do I access the trajectories taken by the model during training?
The text was updated successfully, but these errors were encountered: