Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

gdeval and (py)trec_eval give different ndcg@k values #14

Open
seanmacavaney opened this issue May 16, 2020 · 0 comments
Open

gdeval and (py)trec_eval give different ndcg@k values #14

seanmacavaney opened this issue May 16, 2020 · 0 comments

Comments

@seanmacavaney
Copy link
Contributor

I believe the current configuration will never use gdeval by default for ndcg@k. But when the component is used, it produces different ndcg@k values than pytrec_eval. For instance, on a bm25 run on the COIVID dataset for ndcg@10, we get:

qid	gdeval	pytrec_eval
1	0.3842	0.4074
2	0.4826	0.4826
3	0.3749	0.3987
4	0.0366	0.0550
5	0.3832	0.4174
...
26	0.3963	0.4121
27	0.7602	0.7854
28	0.9148	0.9148
29	0.7467	0.7650
30	0.7305	0.7978
mean	0.5371	0.5618

This seems to be because gdeval uses a variant of ndcg that places higher importance on relevant documents.

This should be parameterized. Something like ndcg_exprel@k?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant