Call compute_all_general_metrics on all requests, not just the last one

[brianwgoldman]

This is a resolution to issue #1989. There are 5 categories of Stats where more data will now be collected:

• compute_efficiency_metrics - how many tokens were sent and how long did the request take.
• compute_finish_reason_metrics - did the request finish due to length, stop tokens, etc.
• compute_truncation_metrics - if the request was truncated
• num_train_trials - no idea why this is even a Stat
• num_references - Count how many references the request had.

To me it seems like the first 3 should be per request and not per instance. The last two are more questionable, but it doesn't seem wrong to do per request.

[percyliang]

This change seems right. Does it affect any of the metric computations in practice?

[brianwgoldman]

This change seems right. Does it affect any of the metric computations in practice?

My read of the code is that anything using BasicMetric in combination with BinaryRankingAdapter, MultipleChoiceSeparateAdapter, or MultipleChoiceCalibratedAdapter will see different values. The following run_spec_function seem to use those:

• msmarco
• blimp - if you don't override the default
• cleva - if the prompt template has "mul_as_gen" in meta set to False.

There are also around 8 more scenarios that accept method as a parameter that could be configured to use those adapters.

I've not done any runs to compare output, as I'm not sure what the best way to do that would be.