Possible bug in SFTTrainer

[jjc10]

Hi folks,

I am running into an issue with the SFTTrainer regarding the neftune_noise_alpha parameter.

If I leave this argument empty and let it default to None, there seems to be an error in the code flow on line 307 of the SFTTrainer. Namely (with added comments):

self._trainer_supports_neftune = hasattr(args, "neftune_noise_alpha") # This evaluates to true when the default None is used
if neftune_noise_alpha is not None and self._trainer_supports_neftune: # this gets skipped because of neftune_noise_alpha is None
            args.neftune_noise_alpha = neftune_noise_alpha
            warnings.warn(
                "You passed a `neftune_noise_alpha` argument to the SFTTrainer, the value you passed will override the one in the `SFTConfig`."
            )
            # self.neftune_noise_alpha is done at Trainer level
elif not self._trainer_supports_neftune: # But this never gets executed and thus self.neftune_noise_alpha is never set
            self.neftune_noise_alpha = neftune_noise_alpha

Later on in the train() at line 448, the error AttributeError: 'SFTTrainer' object has no attribute 'neftune_noise_alpha'

Possible fix:
I believe one fix would be to change line 307 from hasattr(args, "neftune_noise_alpha") to hasattr(args, "neftune_noise_alpha") and neftune_noise_alpha is not None

Additional info:

• OS: Ubuntu 22.04.4
• TRL 0.9.6
• Torch 1.13.0+cu116
• Transformers 4.34.1

Thanks and cheers!

[kashif]

@jjc10 would you like to send a PR?

[kashif]

@jjc10 i fixed it via the above PR

[jjc10]

Sorry just saw your reply @kashif .
Didn't get a chance to open a PR myself this week. I'm reviewing yours!

[dumeixiang]

Hi ,
I've bump into the same issue, tried neftune_noise_alpha=None, 0.1, both trigers the same error: AttributeError: 'SFTTrainer' object has no attribute 'neftune_noise_alpha'
trainer = SFTTrainer( model=model, train_dataset=dataset, neftune_noise_alpha=None, peft_config=peft_config, max_seq_length=max_seq_length, tokenizer=tokenizer, packing=False, formatting_func=format_instruction, args=args, )

[jjc10]

Hi @dumeixiang,
I will pick up the PR next week to submit a fix.
A temporary workaround is to first initialize your trainer without passing neftune_noise_alpha and right after initializing it (that is, before calling sft_trainer.train() or sft_trainer.evaluate()), you manually set sft_trainer.neftune_noise_alpha = None.

Example:

trainer = SFTTrainer(
        model=my_model,
        args=my_config,
        train_dataset=tokenized_dataset['train'],
        eval_dataset=tokenized_dataset['validation'],
        tokenizer=tokenizer,
        data_collator=collator,
    )
    
trainer.neftune_noise_alpha = None
trainer.train()