Fix Sentiment control notebook

[lvwerra]

This is an attempt to fix the instabilities in the controlled sentiment generation.

Changes so far:

• fix logit computation for reward (pipeline changes order natively which can cause the position of the logit to switch)
• In generate kwargs add "eos_token_id": -1

The last point makes sure that the model keeps generating until the max new tokens is reached. I think what happens otherwise is that sometimes the model only generates 1-2 tokens in which case the PPO loss spikes (don't know why, yet).

The last successful run is here.

[HuggingFaceDocBuilderDev]

The documentation is not available anymore as the PR was closed or merged.

[younesbelkada]

Thanks for fixing!

[natolambert]

What's the right way to dev with notebooks and avoid some things like this?

[natolambert]

same as above

[natolambert]

This is the only important line, right?

[natolambert]

Small review, asking about best way to maintain notebooks.
Should we use the NB Review tool for github that is used in huggingface/notebooks?
We could also move the notebooks there?

[lvwerra]

Thanks @natolambert - I actually installed NB Review yesterday but I think we need to open a new PR for it to show the link :) https://app.reviewnb.com/lvwerra/trl/pull/126/

[leoribeiro]

@lvwerra it seems that if we don't set "eos_token_id": -1 the model learn gradually to generate shorter outputs. I think setting "eos_token_id": -1 is not desirable because it makes the model to generate non grammatical text because it forces the generation until the max new tokens is reached. In that way, the model is not useful.

For example, this is a output generated by a RL-trained T5 model with "eos_token_id": -1 for sampling:
Three diaspora-based, prominent African activists discuss the current state of women's rights in the continent. In 2010, the African Union launched a decade-long initiative to promote women's empowerment. In 2010, the African Union launched a decade-long initiative to promote women's capabilities. Organizer Paul Valdais has been named "Inspirational Woman of 2012" by the UK group "Women 4 Africa"TIME: MK:'We really hope to be getting focus on justice at the bottom of the media., despite the

This is the output for the same input example by a T5 model RL-trained without "eos_token_id": -1 for sampling:
African Voices talks with three diaspora-based African activist.

Did you figure it out why T5 model learns to generate shorter inputs with PPO?

[aliwalker]

@lvwerra it seems that if we don't set "eos_token_id": -1 the model learn gradually to generate shorter outputs. I think setting "eos_token_id": -1 is not desirable because it makes the model to generate non grammatical text because it forces the generation until the max new tokens is reached. In that way, the model is not useful.

For example, this is a output generated by a RL-trained T5 model with "eos_token_id": -1 for sampling: Three diaspora-based, prominent African activists discuss the current state of women's rights in the continent. In 2010, the African Union launched a decade-long initiative to promote women's empowerment. In 2010, the African Union launched a decade-long initiative to promote women's capabilities. Organizer Paul Valdais has been named "Inspirational Woman of 2012" by the UK group "Women 4 Africa"TIME: MK:'We really hope to be getting focus on justice at the bottom of the media., despite the

This is the output for the same input example by a T5 model RL-trained without "eos_token_id": -1 for sampling: African Voices talks with three diaspora-based African activist.

Did you figure it out why T5 model learns to generate shorter inputs with PPO?

Hi @leoribeiro ! I'm struggling with this issue when training on a very long context summarization task. Did you find any workarounds?