AlignProp Support for direct reward finetuning

[parthos86]

Is your feature request related to a problem? Please describe.
No. AlignProp makes reward finetuning very fast compared to DDPO (about 25x) becoz of backpropagating the gradients directly from the reward function.

Describe the solution you'd like.
A similar integration to DDPO.
https://github.com/huggingface/diffusers/blob/main/docs/source/en/training/ddpo.md

Describe alternatives you've considered.
There is currently an implementation, however it is not well supported and is not within diffusers pipeline.
https://github.com/mihirp1998/AlignProp/

@mihirp1998

[sayakpaul]

It's more of trl thing really. Cc: @lvwerra @younesbelkada. diffusers is not a training focused library.

[mihirp1998]

I would be happy to do the the integration, although i would need some reference to what format/structure to follow

[sayakpaul]

I am going to close this issue as it better belongs to the trl repo.