Add VL-RewardBench dataset

[TobiasLee]

Hi lmms-eval team,

Thanks for your great project, which accelerates our LMM workflow a lot.

This PR incorporates our recently released VL-RewardBench.

Example evaluation script:

python3 -m accelerate.commands.launch \
    --num_processes=8 \
    -m lmms_eval \
    --model gpt4v \
    --model_args model_version="gpt-4o-mini",modality="image" \
    --tasks vl_rewardbench \
    --batch_size 1 \
    --log_samples \
    --log_samples_suffix 4o_mini_vlrewardbench \
    --output_path ./logs/
    ```

[TobiasLee]

Also, does lmms-eval support majority voting evaluation?
I saw a similar filter implementation, but I'm not sure how we can integrate it with our evaluation loop.
Are there any reference examples using this filter/implementing the voting logic?

[kcz358]

Hi, thank you for your contribution. I think most of the changes are good to go. If you can provide a screenshot of the success evaluation result in this PR would be better.

I think the commit history include some duplicate commits and should be remove to keep the change history clean. I can directly do that on your branch but I need to force push on it so if I can do that I can help you clean it.

For the filtering option, you can try adding

filter_list:
  - name: "xxx"
    filter:
      - function: "majority_vote"

in your yaml file and see if it works. But we directly copy it from lm-eval-harness so we are not sure whether it can work or not. For self-customizing filters, you can check examples such as realworldqa

[TobiasLee]

Here are the saved evaluation results of GPT-4o mini and GPT-4o, which are consistent with our previous results with small variance.

 "results": {
    "vl_rewardbench": {
      "alias": "vl_rewardbench",
      "vlreward_score,none": 0.47855290712392273,
      "vlreward_score_stderr,none": "N/A",
# more details omitted
          }

"results": {
  "vl_rewardbench": {
    "alias": "vl_rewardbench",
    "vlreward_score,none": 0.6168690489841239,
    "vlreward_score_stderr,none": "N/A"
  }
}

For the duplicate commits: you can do it on the branch directly. Thanks a lot!

For the majority voting: Our results indicate that the increased inference computation brings little difference so I think we can leave it for future integration.

[kcz358]

Thanks, I will merge this PR now