LambdarankNDCG objective ignores truncation level

[metpavel]

In /src/objective/rank_objective.hpp: currently, for each query, all pairs of documents contribute to the gradients computation regardless of the truncation level k. However, to correctly optimize for NDCG@k, one should exclude pairs containing both documents beyond the top k (as they don't affect NDCG@k when swapped). Please enforce the check for the truncation level k.

[guolinke]

@metpavel during training, some top docs may be wrong ranking, and not in top-k. In these cases, we still need to consider all docs.

[metpavel]

@guolinke, in fact, we would still consider all documents even if we exclude pairs containing both documents beyond the top-k (as any document will contribute when paired with any other top-k document -- thus any relevant document has a push towards the top of the list no matter what position it is at). Otherwise, we just optimize for NDCG@Full_Rank regardless of the desired truncation level.

[guolinke]

actually, the parameter once named optimize_pos_at_, but I find it actually does not optimize the ndcg@k, and change its name to truncation_level_.

For your solution, exclude pairs containing both documents beyond the top-k, I mis-read it.
I think it is a good solution!
Do you have the interest to create a PR for this change?

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.