Address issue huggingface#1122 (huggingface#1174)

* Address issue huggingface#1122

    Issue [huggingface#1122](huggingface#1122)
    takes care of an inconsistency between `_prepare_packed_dataloader`
    and `_prepare_non_packed_dataloader`

* made attention_mask field in ConstantLengthDataset a tensor

trl/trainer/sft_trainer.py

@@ -402,7 +402,11 @@ def tokenize(element):
                 else:
                     self._dataset_sanity_checked = True
 
-            return {"input_ids": outputs["input_ids"], "attention_mask": outputs["attention_mask"]}
+            return {
+                "input_ids": outputs["input_ids"],
+                "labels": outputs["input_ids"],
+                "attention_mask": outputs["attention_mask"],
+            }
 
         tokenized_dataset = dataset.map(
             tokenize,

trl/trainer/utils.py

@@ -452,6 +452,7 @@ def __iter__(self):
                 yield {
                     "input_ids": torch.LongTensor(example),
                     "labels": torch.LongTensor(example),
+                    "attention_mask": torch.ones(len(example)),
                 }