Is this benchmark being maintained?

[ZhangzihanGit]

Hi,

It seems like the benchmark is not being actively maintained -- the latest session is October 2023. Could you please let me know if you are planning to maintain it as said in your paper?

In addition, I understand that,

KoLA does not allow the local evaluation to prevent test set leakage and unfairly overfitting datasets (cited from the paper)

however, do you have plans to open-source the evaluation data from previous seasons?

Thanks.

[xurh20]

Yes it should be actively maintained.
Not really, as if there's a new model our plan is to test it through all previous seasons data - in which case open-source them will lead to data-leackage.

[xurh20]

Hi, just fyi that the newest s3 test result is updated on the website. You can have a check if you are interested, thx!