Push down sort through eval

[qianheng-aws]

Description

Add a rule EvalPushDown.PUSH_DOWN_SORT, this rule will push down limit under eval. Thus, sort has chance to be pushed down into TableScanBuilder later.

For now, it only supports push specific sort and eval, which has restriction:

1. The expression in sort and replaced expression are both ReferenceExpression.
2. No internal reference in eval.

As described in this RFC #2864 , we may have chance to extend it.

e.g.

POST _plugins/_ppl/_explain
{
  "query": """
    source=opensearch_dashboards_sample_data_flights | eval newFlightMin = FlightTimeMin | sort newFlightMin |  fields newFlightMin 
  """
}

# Before optimization
{
  "root": {
    "name": "ProjectOperator",
    "description": {
      "fields": "[newFlightMin]"
    },
    "children": [
      {
        "name": "SortOperator",
        "description": {
          "sortList": {
            "newFlightMin": {
              "sortOrder": "ASC",
              "nullOrder": "NULL_FIRST"
            }
          }
        },
        "children": [
          {
            "name": "EvalOperator",
            "description": {
              "expressions": {
                "newFlightMin": "FlightTimeMin"
              }
            },
            "children": [
              {
                "name": "OpenSearchIndexScan",
                "description": {
                  "request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":200,"timeout":"1m"}, searchDone=false)"""
                },
                "children": []
              }
            ]
          }
        ]
      }
    ]
  }
}

# After optimization
{
  "root": {
    "name": "ProjectOperator",
    "description": {
      "fields": "[newFlightMin]"
    },
    "children": [
      {
        "name": "EvalOperator",
        "description": {
          "expressions": {
            "newFlightMin": "FlightTimeMin"
          }
        },
        "children": [
          {
            "name": "OpenSearchIndexScan",
            "description": {
              "request": """OpenSearchQueryRequest(indexName=opensearch_dashboards_sample_data_flights, sourceBuilder={"from":0,"size":10000,"timeout":"1m","sort":[{"FlightTimeMin":{"order":"asc","missing":"_first"}}]}, searchDone=false)"""
            },
            "children": []
          }
        ]
      }
    ]
  }
}

Related Issues

Resolves #2904

Check List

• New functionality includes testing.
• New functionality has been documented.
• New functionality has javadoc added.
• New functionality has a user manual doc added.
• API changes companion pull request created.
• Commits are signed per the DCO using --signoff.
• Public documentation issue/PR created.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

[ykmr1224]

I think you can return sort at the end of for block once.

[qianheng-aws]

Then we need a flag to indicate returning sort or not for each iteration.

It could be simplified by pattern matching with java version>=14. But since we also need to support java11 for 2.x, I don't use that feature in order to keep code align.

[ykmr1224]

Is it right to return ref as default value?

[qianheng-aws]

Logically, it's right I think, though the name of replacedExpr may be a little bit misleading. It includes 2 cases:

1. the ref is produced by eval operator, then it needs to be replaced
2. the ref isn't produced by eval operator, then we don't need to replace it. That's why I use the default value ref.

How about changing the name of replacedExpr to newExpr?

[ykmr1224]

Looks this Lambda is too large. Please extract to a method or a class. And extract some portion to reduce the size of method for readability.

[ykmr1224]

Please split the method for each test case.