Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

(Refactor)[Nereids] Combine operator and plan #10786

Merged
merged 8 commits into from
Jul 13, 2022

Conversation

924060929
Copy link
Contributor

@924060929 924060929 commented Jul 12, 2022

Proposed changes

Issue Number: no issue

Problem Summary:

in #9755, we split plan into plan & operator, but in subsequent development, we found the rule became complex and counter intuition:

  1. we must create an operator instance, then wrap a plan by the operator type.
  2. relational algebra(operator) not contains children

e.g.

logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.operator.getProjects(), project.children(), project);
    LogicalProject op = new LogicalProject(flatBoundStar(boundSlots));
    // wrap a plan
    return new LogicalUnaryPlan(op, project.child());
})

after combine operator and plan, the code become to:

logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.getProjects(), project.children(), project);
    return new LogicalProject(flatBoundStar(boundSlots), project.child());
})

Originally, we thought it would be convenient for Memo.copyIn() after split plan & operator, because Memo don't known how to re-new the plan(assembling child plan in the children groups) by the plan type. So plan must provide the withChildren() abstract method to assembling children. The less plan type, the lower code cost we have(logical/physical with leaf/unary/binary plan, about 6 plans, no concrete plan e.g. LogicalAggregatePlan).

But the convenient make negative effect that difficult to understand, and people must known the concept then can develop some new rules, and rule become ugly. So we combine the plan & operator, make the rule as simple as possible, the negative effect is we must overwrite some withXxx for all concrete plan, e.g. LogicalAggregate, PhysicalHashJoin.

Checklist(Required)

  1. Does it affect the original behavior: No
  2. Has unit tests been added: No Need
  3. Has document been added or modified: No Need
  4. Does it need to update dependencies: No
  5. Are there any changes that cannot be rolled back: No

@924060929 924060929 changed the title (Refactor)[Nereids] Combine op plan (Refactor)[Nereids] Combine operator and plan Jul 12, 2022
@github-actions github-actions bot added area/nereids area/planner Issues or PRs related to the query planner labels Jul 12, 2022
public Operator getOperator() {
return op;
public Plan getPlan() {
return plan;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need to set group's logical properties to plan

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the plan copyIn to Memo, the plan will be withLogicalProperties and withChildren(GroupPlan... childrens):

private Plan replaceChildrenToGroupPlan(Plan plan, List<Group> childrenGroups) {
        List<Plan> groupPlanChildren = childrenGroups.stream()
                .map(group -> new GroupPlan(group))
                .collect(ImmutableList.toImmutableList());
        LogicalProperties logicalProperties = plan.getLogicalProperties();
        return plan.withChildren(groupPlanChildren)
            .withLogicalProperties(Optional.of(logicalProperties));
}

so the plan in the memo contains the logicalProperties same as the logicalProperties in the groupExpression, and replace GroupPlan as children

@@ -158,7 +158,8 @@ private enum DateLiteralType {
DATEV2(3);

private final int value;
private DateLiteralType(int value) {

DateLiteralType(int value) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why do this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fix checkstyle. and private enum DateLiteralType's constructor is also private, so can be omitted


@Override
public Plan withLogicalProperties(Optional<LogicalProperties> logicalProperties) {
return new UnboundRelation(nameParts, Optional.empty(), logicalProperties);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

groupexpression should use original one?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

when the plan is changed, the groupExpression must be clear to tell the memo: I create a new plan, don't skip copyIn.

@morningman
Copy link
Contributor

This PR didn't pass the p0 test, because there are some other issue not related to this PR.
So I skip p0 for this PR.

Copy link
Contributor

@EmmyMiao87 EmmyMiao87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@EmmyMiao87 EmmyMiao87 merged commit e78cca1 into apache:master Jul 13, 2022
eldenmoon pushed a commit to eldenmoon/incubator-doris that referenced this pull request Aug 1, 2022
in apache#9755, we split plan into plan & operator, but in subsequent development, we found the rule became complex and counter intuition: 
1. we must create an operator instance, then wrap a plan by the operator type.
2. relational algebra(operator) not contains children 

e.g.
```java
logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.operator.getProjects(), project.children(), project);
    LogicalProject op = new LogicalProject(flatBoundStar(boundSlots));
    // wrap a plan
    return new LogicalUnaryPlan(op, project.child());
})
```

after combine operator and plan, the code become to:
```java
logicalProject().then(project -> {
    List<NamedExpression> boundSlots =
        bind(project.getProjects(), project.children(), project);
    return new LogicalProject(flatBoundStar(boundSlots), project.child());
})
```

Originally, we thought it would be convenient for `Memo.copyIn()` after split plan & operator, because Memo don't known how to re-new the plan(assembling child plan in the children groups) by the plan type. So plan must provide the `withChildren()` abstract method to assembling children. The less plan type, the lower code cost we have(logical/physical with leaf/unary/binary plan, about 6 plans, no concrete plan e.g. LogicalAggregatePlan). 

But the convenient make negative effect that difficult to understand, and people must known the concept then can develop some new rules, and rule become ugly. So we combine the plan & operator, make the rule as simple as possible, the negative effect is we must overwrite some withXxx for all concrete plan, e.g. LogicalAggregate, PhysicalHashJoin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/nereids area/planner Issues or PRs related to the query planner kind/skip-p0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants