[Feature] Added CompassArena-SubjectiveBench with Bradley-Terry Model

[acylam]

Motivation

Adapt the Bradley-Terry rating system from FastChat to the subjective evaluation setting, but replacing human evaluators with LLM-as-a-judge.

Modification

• Added the Bradley-Terry rating method for subjective evaluation.
• Added pairwise_bt_judge for both singleturn and multiturn evaluation for compass_arena_subjective_bench.
• Added the keep_preds argument to the init method of LMEvaluator for an option to keep LLM predictions (from the inference stage) when saving judge responses during the evaluation stage. This is useful when the postprocessor needs to calculate metadata or metrics based on each LLM's predictions (e.g. response length).
• Added a step at the end of CompassArenaBradleyTerrySummarizer.summarize to fit another BT model with the first base_model and combining matches from all subsets to produce a single rating for each judge_model-LLM combination. The ratings are returned in this format: {'CompassArenaSubjBenchBradleyTerry': {'judge_model': {'model_1': xxx, 'model_2': xxx}}}
• See README for more information.

BC-breaking (Optional)

No breaking changes.

Use cases (Optional)

Use cases and examples shown in README

Checklist

Before PR:

• [x ] Pre-commit or other linting tools are used to fix the potential lint issues.
• [x ] Bug fixes are fully covered by unit tests, the case that causes the bug should be added in the unit tests.
• [x ] The modification is covered by complete unit tests. If not, please add more unit test to ensure the correctness.
• [x ] The documentation has been modified accordingly, like docstring or example tutorials.