5-fold cross validation

[DJHeisenberg01]

Hi again Lambeq community,

I am currently testing both the quantum pipeline using jax and the classical pipeline shown in the examples using my own dataset, rather than asking for help on a piece of code, I wanted to know if it would be possible to perform a 5-fold cross validation in both pipelines and what approach I should use, as I cannot find a way to make predictions and evaluations of the predictions that is included in the trainer description, probably due to not reading the library carefully.

Thanks for your help

[dimkart]

Hi, you can always do cross-validation by including your model, datasets and trainer in a loop for $k$ folds. Be sure to record the results at the end of the loop.

Now, regarding "making predictions and evaluations of the predictions", the tutorials explain the process. You will create your evaluation_functions dictionary, e.g.:

def accuracy(y_hat, y):
    return torch.sum(torch.eq(torch.round(sig(y_hat)), y))/len(y)/2  # half due to double-counting

eval_metrics = {"acc": accuracy}

and you will pass it to the trainer (with evaluate_functions argument). After fit is complete, you can collect the validation results by trainer.val_results['acc'].

Let me know if you need more help.

[DJHeisenberg01]

Okay, for example if i divide my circuits in 5 parts, with 4 parts i create train Dataset and the one remaining i create test Dataset, then i run trainer.fit with these datasets and for every fold after training i use trainer.val_results['acc'] for every results, right?

[dimkart]

Yes that's right, with the difference that in cross-validation you divide your train dataset into 5 parts, not the entire data. Your test dataset remain untouched to evaluate on it a single time at the end (that's the usual practice at least).

[dimkart]

One more clarification, in case it's not obvious. You use cross-validation to test different model configurations and select the one that performs better across all different folds. After that, you will have to retrain your selected model on the entire training set. Then you apply this model on your kept-out test set.