Avoid negative scores with cross_fields type

[jtibshirani]

The cross_fields scoring type can produce negative scores when some documents
are missing fields. When blending term document frequencies, we take the maximum
document frequency across all fields. If one field appears in fewer documents
than another, this means that its IDF can become negative. This is because IDF
is calculated as Math.log(1 + (docCount - docFreq + 0.5) / (docFreq + 0.5))

This change adjusts the docFreq for each field to Math.min(docCount, docFreq)
so that the IDF can never become negative. It makes sense that the term document
frequency should never exceed the number of documents containing the field.

Fixes #44700

[elasticsearchmachine]

Hi @jtibshirani, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @jtibshirani, I've updated the changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Hi @jtibshirani, I've created a changelog YAML for you.

[jtibshirani]

This query's score calculation is complex and can be hard to reason about. I liked this change because it doesn't affect the non-broken case at all. It only changes the score if docCount < docFreq, which would have resulted in a negative IDF and a broken score anyways.

[romseygeek]

I think @jpountz will want to have a look as well but this makes sense to me.

[jtibshirani]

Thanks for the review @romseygeek.

[jtibshirani]

@jpountz would you be up for reviewing this? Thanks in advance!

[jpountz]

Thanks @jtibshirani for the ping. This looks like the smallest change we could do that would fix the bug, but I worry that it goes against the intent of cross_fields. cross_fields's documentation says Treats fields with the same analyzer as though they were one big field and my understanding is that the implementation tries to achieve this by forging term statistics in a way that gives terms (almost) the same IDF on all fields. If we start adjusting docFreq then we're possibly giving very different IDFs to different fields for the same term.

Would it make more sense to adjust docCount to take the max across all fields, which should in-turn help guarantee that docFreq is always less than docCount? I know that different fields may be getting very different IDFs today because they have different docCounts but my understanding is that it is due to the fact that the cross_fields implementation overlooked sparse fields and it was not intentional.

[jtibshirani]

@jpountz thanks for looking. I considered adjusting docCount instead, but I could not find a way to change that value, since it is not part of TermStates and is loaded from IndexReader. Only docFreq can be adjusted... here's the javadoc from TermQuery:

  /**
   * Expert: constructs a TermQuery that will use the provided docFreq instead of looking up the
   * docFreq against the searcher.
   */
  public TermQuery(Term t, TermStates states) { ... }

We could attempt a larger refactor, but it seems risky and not the right investment.

As you mentioned, we overlooked the sparse case so the statement "Treats fields with the same analyzer as though they were one big field" was never accurate. I think our options are (1) consider a bigger, principled change to fix the sparse case, (2) throw an exception in the sparse case saying it's not supported, or (3) make a targeted fix like this one. I would vote for (3) as the most user friendly and practical option.

[jpountz]

Thanks @jtibshirani for the additional context. Now that I better understand what it would take to implement a correct fix, I agree with your suggestion of going with option 3, essentially further redirecting users from the cross_fields mode of multi_match to the combined_fields query.

[jpountz]

Given the discussion on fixing docCount vs. docFreq, I wonder if we should make this warning stronger and point out that scores may not only be hard to interpret (which covers e.g. the way it tries to prefer the most likely field) but also incorrect in case of fields that have different statistics.

[jtibshirani]

👍