[Gen AI] allenai Model usage.

[head-iie-vnr]

• Create custom training data
• Train on the custom training data.

[head-iie-vnr]

When I used batch_size of 2 it crashed memory.
Even after briniging down the initial memory state to be 13.5 GB free (1.6GB use), it still crashed hitting the 16GB upper limit.

When I reduced the batch_size to 1, it was manageble.

Special observation: Each training iteration was taking 20seconds and the same can be observed in the Graph Heart Beat.
The lowest point is the start of new step(iteration)

[head-iie-vnr]

The training data contains
40 questions & answers

Original context text:
contains 900 words
total sentences are 50.

[head-iie-vnr]

Code: Step-by-Step Explanation

1. Importing Required Libraries:

   import json
   from transformers import AutoTokenizer, AutoModelForQuestionAnswering, Trainer, TrainingArguments
   from datasets import DatasetDict, Dataset

   • json: For loading the custom dataset from a JSON file.
   • transformers: For using the Hugging Face library to handle tokenization, model loading, and training.
   • datasets: For managing the dataset in a format compatible with Hugging Face's transformers.

2. Loading Custom Dataset:

   def load_custom_dataset(file_path):
       with open(file_path, 'r') as f:
           dataset_dict = json.load(f)
       return dataset_dict
   
   # Assuming your dataset file path is 'custom_qa_dataset.json'
   custom_dataset = load_custom_dataset('custom_qa_dataset.json')

   • This function reads a JSON file containing the custom QA dataset and loads it into a Python dictionary.

3. Converting to SQuAD Format:

   def convert_to_squad_format(custom_dataset):
       contexts = []
       questions = []
       answers = []
       
       for data in custom_dataset["data"]:
           for paragraph in data["paragraphs"]:
               context = paragraph["context"]
               for qa in paragraph["qas"]:
                   question = qa["question"]
                   for answer in qa["answers"]:
                       contexts.append(context)
                       questions.append(question)
                       answers.append({
                           "text": answer["text"],
                           "answer_start": answer["answer_start"]
                       })
       
       return {
           "context": contexts,
           "question": questions,
           "answers": answers
       }
   
   squad_format_dataset = convert_to_squad_format(custom_dataset)

   • This function converts the custom dataset into a format similar to the SQuAD dataset format.
   • It extracts contexts, questions, and answers into separate lists and returns a dictionary with these lists.

4. Creating Hugging Face Dataset:

   dataset = DatasetDict({"train": Dataset.from_dict(squad_format_dataset)})

   • This creates a Hugging Face Dataset from the structured data and puts it into a DatasetDict under the "train" key.

5. Loading the Tokenizer and Model:

   model_name = "allenai/longformer-base-4096"
   tokenizer = AutoTokenizer.from_pretrained(model_name)
   model = AutoModelForQuestionAnswering.from_pretrained(model_name)

   • Loads the tokenizer and model for the Longformer, which is capable of handling long contexts up to 4096 tokens.

6. Tokenizing the Dataset:

   def preprocess_function(examples):
       inputs = tokenizer(
           examples["question"],
           examples["context"],
           max_length=4096,
           truncation=True,
           padding="max_length",
           return_offsets_mapping=True,
       )
       offset_mapping = inputs.pop("offset_mapping")
       start_positions = []
       end_positions = []
   
       for i, answer in enumerate(examples["answers"]):
           start_char = answer["answer_start"]
           end_char = start_char + len(answer["text"])
           sequence_ids = inputs.sequence_ids(i)
   
           # Find the start and end of the context
           idx = 0
           while sequence_ids[idx] != 1:
               idx += 1
           context_start = idx
           while sequence_ids[idx] == 1:
               idx += 1
           context_end = idx - 1
   
           # If the answer is out of the context, label it (0, 0)
           if not (offset_mapping[i][context_start][0] <= start_char and offset_mapping[i][context_end][1] >= end_char):
               start_positions.append(0)
               end_positions.append(0)
           else:
               start_idx = context_start
               while offset_mapping[i][start_idx][0] <= start_char:
                   start_idx += 1
               start_positions.append(start_idx - 1)
   
               end_idx = context_end
               while offset_mapping[i][end_idx][1] >= end_char:
                   end_idx -= 1
               end_positions.append(end_idx + 1)
   
       inputs["start_positions"] = start_positions
       inputs["end_positions"] = end_positions
       return inputs
   
   tokenized_datasets = dataset.map(preprocess_function, batched=True, batch_size=2)

   • The preprocess_function tokenizes the questions and contexts.
   • It calculates the start and end positions of the answers in the tokenized context.
   • dataset.map(preprocess_function, batched=True, batch_size=2) applies the preprocessing function to the dataset in batches of size 2.

7. Setting Up Training Arguments:

   training_args = TrainingArguments(
       output_dir="./results",
       evaluation_strategy="epoch",
       learning_rate=2e-5,
       per_device_train_batch_size=1,  # Reduced batch size
       per_device_eval_batch_size=1,  # Reduced batch size
       num_train_epochs=3,
       weight_decay=0.01,
       gradient_accumulation_steps=8,  # Use gradient accumulation
       fp16=True,  # Enable mixed precision training
   )

   • Configures the training parameters, such as output directory, learning rate, batch size, number of epochs, gradient accumulation, and mixed precision training.

8. Initializing the Trainer:

   trainer = Trainer(
       model=model,
       args=training_args,
       train_dataset=tokenized_datasets["train"],
       eval_dataset=tokenized_datasets["train"],
   )

   • Initializes the Trainer with the model, training arguments, and the tokenized dataset for training and evaluation.

9. Training the Model:

   trainer.train()

   • Trains the model using the specified training arguments and dataset.

10. Saving the Fine-Tuned Model:

    model.save_pretrained("./fine-tuned-longformer")
    tokenizer.save_pretrained("./fine-tuned-longformer")

    • Saves the fine-tuned model and tokenizer to the specified directory.

Summary

The code loads a custom QA dataset, converts it to a format compatible with Hugging Face's transformers library, tokenizes the data, sets up training parameters, trains the Longformer model on the dataset, and finally saves the fine-tuned model and tokenizer.

[head-iie-vnr]

The results are not satisfactory

Question: When was Gandhi born?
Answer: was born
Score: 0.0077830287627875805
Question: Where was Gandhi born?
Answer: was born

[head-iie-vnr]

Trying different model in Issue#14