Question about the loss function and reference model

[Richar-Du]

Thanks for your awesome work! I'm studying your code and want to implement it in my system. I have the following two questions:

1. When calculating the delta, I wonder why the discount factor gamma times nextvalues instead of rewards[t+1]? I think the cumulated reward has no relation to the value.
   https://github.com/lvwerra/trl/blob/44fb7326fc2440756f27e38be5220dd668fc92bc/trl/ppo.py#L237

2. I notice that in other implementations, the old policy network updates as well (but slower than the active policy network). In your implementation, the old policy network doesn't update and always keeps the same as the checkpoint of GPT-2. Am I right?

Could you please explain these questions? Thanks again :)

[lvwerra]

Regarding 1: see equation (11) in https://arxiv.org/abs/1506.02438 and 2) yes you are correct.