Spearman as metric

[fdzr]

I'm confused about how to create a metric (function or program) that accumulates predictions and gold data to compute the Spearman correlation. My confusion stems from the concept of accuracy, which evaluates each individual prediction. I want to ensure that all predictions are considered collectively at the end of the evaluation process.

I need help!

[okhat]

You may need to make your individual examples contain entire batches of sub-examples :D

[fdzr]

@okhat any clue on how to code this?

[okhat]

trainset = [dspy.Example(examples=[...])]

[fdzr]

@okhat thank you so much!!!!

[fdzr]

@okhat When using the MIPROV2 optimizer, I need to provide a metric as a parameter, but I'm uncertain about the format or structure of the metric. Any clue considering that the target is optimize Spearman?

[okhat]

You need a batch metric! That takes example, pred, trace=None and looks inside example.batch for a batch of things and in pred.batch for a batch of predictions.

[fdzr]

@okhat Essentially, the Spearman score is being reported. The MiproV2 optimizer, operating in the background, recognizes that a higher score indicates a better program. It then utilizes these weights to identify and report the optimal program, correct?

[okhat]

Yeah