Doing Counterfactual Experience Replay

[nikhil-pitta]

I would like to try and do some counterfactual experience replay (specifically I want to use PPO and do CER on the critic). Is there something that exposes the replay buffer or something that allows me to add custom trajectories while training somehow?

Also, how do I access the trajectories taken by the model during training?

[Aequatio-Space]

You can try overriding the postprocess_fn in PPOTOrchPolicy. In the MARLlib, such an example may be:

MARLlib/marllib/marl/algos/core/CC/coma.py

Lines 116 to 125 in 368c617

	COMATorchPolicy = A3CTorchPolicy.with_updates(
	name="COMATorchPolicy",
	get_default_config=lambda: A2C_CONFIG,
	postprocess_fn=centralized_critic_postprocessing,
	loss_fn=central_critic_coma_loss,
	extra_action_out_fn=coma_model_value_predictions,
	mixins=[
	CentralizedValueMixin
	])

the signature for postprocess_fn is fixed, which is

postprocess_fn(
policy: Policy, sample_batch: SampleBatch, 
other_agent_batches: Optional[Dict[AgentID, SampleBatch]] = None, 
episode: Optional[MultiAgentEpisode] = None
) -> SampleBatch

As displayed in the build_policy_class in https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy_template.py#L40-L43

You can manipulate the SampleBatch instance provided in the signature to add custom trajectories.

[nikhil-pitta]

@Aequatio-Space Is this function called before or after both the policy gradient and value function gradient update in ippo or in between?

[Aequatio-Space]

@nikhil-pitta it is called before both the policy gradient and value function gradient. The pipeline is basically: extra_action_out_fn → postprocess_fn → loss_fn → compute_gradients → apply_gradients.

[nikhil-pitta]

@Aequatio-Space Thanks so much for the info! We would like to add counterfactual experience to DDPG before adding to the replay buffer or when pulling from the replay buffer to evaluate. Specifically, we want to augment our current step/collected experiences and add to the replay buffer for the algorithm to use. What callback function could we do to use this, and would it be the same for every offline policy? If not a callback, what would you suggest we do?

[Aequatio-Space]

@nikhil-pitta You mentioned "augment our current step/collected experiences and add to the replay buffer", and it sounds exactly what postprocess_fn do, as our earlier discussion. This extra function applies to all MARLlib algorithms since the interface is identical.

[nikhil-pitta]

@Aequatio-Space Thank you for the response, and yes that function is exactly what I need! I tried using it for PPO by overriding the postprocess_fn for the IPPOTorchPolicy (https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/IL/ppo.py; line 31).

I wanted to do the same thing for IQL, and saw this file https://github.com/Replicable-MARL/MARLlib/blob/master/marllib/marl/algos/core/VD/iql_vdn_qmix.py, but I was unsure how to add a postprocess_fn or any other callback because it looks like a new policy is written up within this file that doesn't accept callback functions like the PPOTorchPolicy.

Do you know what I can do to add callback functionality for IQL?

[Aequatio-Space]

@nikhil-pitta Note that JointQPolicy inherits the Policy class, which has a function postprocess_trajectory.
https://github.com/ray-project/ray/blob/55fc0710d8472a9abaf244ed6567eb3b13136531/rllib/policy/policy.py#L361-L366.
Directly overriding this function may help.

[nikhil-pitta]

@Aequatio-Space Thanks for the response again! So now I am trying use IQL, but an issue I'm facing is that I want each of my agents to have separate policies, but I run into the exception ValueError("joint Q learning does not support individual function"), which comes from setting share_policy="individual."

This is not my desired behavior because I would like each agent to have their own policies, which is how I expected IQL to be implemented. Is there any way to get around this?

[Aequatio-Space]

I remember joint Q learning supports share_policy=all, you can see the related logic at here.

MARLlib/marllib/marl/algos/run_vd.py

Lines 105 to 118 in 368c617

	elif exp_info["share_policy"] == "group":
	groups = policy_mapping_info["team_prefix"]
	if len(groups) == 1:
	obs_space = Tuple([GymDict(space_obs)] * n_agents)
	act_space = Tuple([space_act] * n_agents)
	if not policy_mapping_info["all_agents_one_policy"]:
	raise ValueError("in {}, policy can not be shared".format(map_name))
	grouping = {"group_all_": agent_name_ls}
	else:
	raise ValueError("joint Q learning does not support group function")
	elif exp_info["share_policy"] == "individual":
	raise ValueError("joint Q learning does not support individual function")
	else:
	raise ValueError("wrong share_policy {}".format(exp_info["share_policy"]))

Try to adapt the code under this setting.