FEAT: [SFTTrainer] Add eval_packing

[younesbelkada]

Fixes: #1343

Adds the possibility for users to enable/disable packing for the eval_dataset using a ternary logic (if None -> packing else eval_packing)

cc @lvwerra @kashif

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[federicoBetti]

Hi @younesbelkada ,
i'm using eval_packing = False to better compute metrics on the evaluation dataset.
The problem is when using it together with batch_eval_metrics, the Trainer crashes when it tries to delete the tensors of losses, logits, labels, inputs.

            if self.args.batch_eval_metrics:
                if self.compute_metrics is not None and logits is not None and labels is not None:
                    is_last_step = self.accelerator.gradient_state.end_of_dataloader
                    if args.include_inputs_for_metrics:
                        metrics = self.compute_metrics(
                            EvalPrediction(predictions=logits, label_ids=labels, inputs=inputs),
                            compute_result=is_last_step,
                        )
                    else:
                        metrics = self.compute_metrics(
                            EvalPrediction(predictions=logits, label_ids=labels),
                            compute_result=is_last_step,
                        )

                del losses, logits, labels, inputs

This is due to the fact that, when packing is False, the labels are not kept in the input tensor, thus loss and labels after the prediction steps are None. And here the code crashes because losses is not defined.
UnboundLocalError: local variable 'losses' referenced before assignment
Is there something I'm missing to make it work properly?

Using transformers==4.41.2

---

Update
With transformers==4.42.4 this problem is solved. Although, in the same code snippet I shared, labels is still None, thus the compute_metrics function is not called.
This is happening because input doesn't have labels in it from the eval_dataset, if eval_packing is False.