Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure during creation of plan fragments: Cannot overwrite partitioning with SINGLE (currently set to Optional[HASH]) #14317

Closed
phd3 opened this issue Sep 27, 2022 · 4 comments · Fixed by #14872
Closed

Failure during creation of plan fragments: Cannot overwrite partitioning with SINGLE (currently set to Optional[HASH]) #14317

phd3 opened this issue Sep 27, 2022 · 4 comments · Fixed by #14872
Labels
bug Something isn't working

Comments

@phd3
Copy link
Member

phd3 commented Sep 27, 2022 •
edited by findepi
Loading

Saw an issue which is reproducible with the following sequence

CREATE TABLE f1 (k1 bigint, k2 bigint);
CREATE TABLE f2_p (k1 bigint, k2 bigint) WITH (partitioned_by = array['k2']);
CREATE TABLE f3 (k1 bigint, k2 bigint);
INSERT INTO f1 VALUES (5, 6);
INSERT INTO f2_p VALUES (5, 6);
INSERT INTO f3 VALUES (5, 6);


WITH t1 AS (
  SELECT 
    k1 
  FROM (
    SELECT 
      k1,
      ROW_NUMBER() OVER (PARTITION BY k1) AS some_rank
    FROM f1
  )
  WHERE 
    some_rank = 1
  ),
t2 AS (
  SELECT k1 FROM f2_p WHERE k2 - 5 = 8)
SELECT 
  t3.k1 FROM f3 AS t3
  INNER JOIN t1 on t3.k1 = t1.k1 
  INNER JOIN t2 ON t3.k1 = t2.k1;

Query Plan

Output[k1]
│   Layout: [k1:bigint]
│   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
└─ RemoteExchange[GATHER]
   │   Layout: [k1:bigint]
   │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
   └─ InnerJoin[(""k1"" = ""k1_5"")][$hashvalue, $hashvalue_13]
      │   Layout: [k1:bigint]
      │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │   Distribution: PARTITIONED
      │   maySkipOutputDuplicates = false
      ├─ InnerJoin[(""k1"" = ""k1_0"")][$hashvalue, $hashvalue_11]
      │  │   Layout: [k1:bigint, $hashvalue:bigint]
      │  │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │  │   Distribution: PARTITIONED
      │  │   maySkipOutputDuplicates = false
      │  ├─ RemoteExchange[REPARTITION][$hashvalue]
      │  │  │   Layout: [k1:bigint, $hashvalue:bigint]
      │  │  │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │  │  └─ Project[]
      │  │     │   Layout: [k1:bigint, $hashvalue_10:bigint]
      │  │     │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │  │     │   $hashvalue_10 := combine_hash(bigint '0', COALESCE(""$operator$hash_code""(""k1""), 0))
      │  │     └─ Values
      │  │            Layout: [k1:bigint]
      │  │            Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │  └─ LocalExchange[HASH][$hashvalue_11] (""k1_0"")
      │     │   Layout: [k1_0:bigint, $hashvalue_11:bigint]
      │     │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │     └─ Project[]
      │        │   Layout: [k1_0:bigint, $hashvalue_12:bigint]
      │        │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      │        │   $hashvalue_12 := combine_hash(bigint '0', COALESCE(""$operator$hash_code""(""k1_0""), 0))
      │        └─ Values
      │               Layout: [k1_0:bigint]
      │               Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
      └─ LocalExchange[HASH][$hashvalue_13] (""k1_5"")
         │   Layout: [k1_5:bigint, $hashvalue_13:bigint]
         │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
         └─ RemoteExchange[REPARTITION][$hashvalue_14]
            │   Layout: [k1_5:bigint, $hashvalue_14:bigint]
            │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
            └─ Project[]
               │   Layout: [k1_5:bigint, $hashvalue_15:bigint]
               │   Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}
               │   $hashvalue_15 := combine_hash(bigint '0', COALESCE(""$operator$hash_code""(""k1_5""), 0))
               └─ Values
                      Layout: [k1_5:bigint]
                      Estimates: {rows: 0 (0B), cpu: 0, memory: 0B, network: 0B}

Failure in PlanFragmenter

Caused by: java.lang.IllegalStateException: Cannot overwrite partitioning with SINGLE (currently set to Optional[HASH])
	at com.google.common.base.Preconditions.checkState(Preconditions.java:821)
	at io.trino.sql.planner.PlanFragmenter$FragmentProperties.setSingleNodeDistribution(PlanFragmenter.java:446)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitValues(PlanFragmenter.java:351)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitValues(PlanFragmenter.java:202)
	at io.trino.sql.planner.plan.ValuesNode.accept(ValuesNode.java:131)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:37)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:21)
	at io.trino.sql.planner.plan.PlanVisitor.visitProject(PlanVisitor.java:39)
	at io.trino.sql.planner.plan.ProjectNode.accept(ProjectNode.java:82)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitExchange(PlanFragmenter.java:359)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitExchange(PlanFragmenter.java:202)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:243)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:37)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:21)
	at io.trino.sql.planner.plan.PlanVisitor.visitJoin(PlanVisitor.java:99)
	at io.trino.sql.planner.plan.JoinNode.accept(JoinNode.java:316)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitExchange(PlanFragmenter.java:359)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitExchange(PlanFragmenter.java:202)
	at io.trino.sql.planner.plan.ExchangeNode.accept(ExchangeNode.java:243)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:37)
	at io.trino.sql.planner.plan.SimplePlanRewriter.visitPlan(SimplePlanRewriter.java:21)
	at io.trino.sql.planner.plan.PlanVisitor.visitJoin(PlanVisitor.java:99)
	at io.trino.sql.planner.plan.JoinNode.accept(JoinNode.java:316)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.rewrite(SimplePlanRewriter.java:81)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.lambda$defaultRewrite$0(SimplePlanRewriter.java:72)
	at com.google.common.collect.ImmutableList.forEach(ImmutableList.java:422)
	at io.trino.sql.planner.plan.SimplePlanRewriter$RewriteContext.defaultRewrite(SimplePlanRewriter.java:72)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitOutput(PlanFragmenter.java:266)
	at io.trino.sql.planner.PlanFragmenter$Fragmenter.visitOutput(PlanFragmenter.java:202)
	at io.trino.sql.planner.plan.OutputNode.accept(OutputNode.java:83)
	at io.trino.sql.planner.plan.SimplePlanRewriter.rewriteWith(SimplePlanRewriter.java:31)
	at io.trino.sql.planner.PlanFragmenter.createSubPlans(PlanFragmenter.java:125)
	at io.trino.execution.SqlQueryExecution.doPlanQuery(SqlQueryExecution.java:473)
	at io.trino.execution.SqlQueryExecution.planQuery(SqlQueryExecution.java:450)
	at io.trino.execution.SqlQueryExecution.start(SqlQueryExecution.java:391)
	at io.trino.execution.SqlQueryManager.createQuery(SqlQueryManager.java:249)
	at io.trino.dispatcher.LocalDispatchQuery.lambda$startExecution$7(LocalDispatchQuery.java:143)
	at io.trino.$gen.Trino_testversion____20220927_032348_71.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
	at java.base/java.lang.Thread.run(Thread.java:833)

The issue is reproducible on current master, as well as as far back as 352
@phd3 phd3 added the bug Something isn't working label Sep 27, 2022
@findepi findepi changed the title Failure during creation of plan fragments Failure during creation of plan fragments: Cannot overwrite partitioning with SINGLE (currently set to Optional[HASH]) Sep 27, 2022
@findepi
Copy link
Member

findepi commented Sep 27, 2022

@phd3 Might be a duplicate of #10887

@phd3
Copy link
Member Author

phd3 commented Sep 27, 2022

Might be a duplicate of #10887

yeah, may be. Although here bucketed tables are not involved, but root cause may be the same.

cc @martint @kasiafi is the plan "wrong"? It seems that for Values, the fragmenter always tries to add SINGLE partitioning - which might go wrong if there's a join enforcing hash.

@findepi
Copy link
Member

findepi commented Sep 27, 2022

cc @sopel39 @dain @raunaqmorarka

@martint
Copy link
Member

martint commented Nov 2, 2022

It happens because this subplan:

               ├─ Project[]
               │  │   Layout: [a:bigint]
               │  └─ FilterProject[filterPredicate = false]
               │     │   Layout: [a:bigint]
               │     └─ RowNumber[partitionBy = [a], limit = 1]
               │        │   Layout: [a:bigint, row_number:bigint]
               │        │   row_number := row_number()
               │        └─ RemoteExchange[type = REPARTITION]
               │           │   Layout: [a:bigint]
               │           └─ TableScan[table = hive:tpch:t]
               │                  Layout: [a:bigint]
               │                  a := a:bigint:REGULAR

is being rewritten to:

               ├─ Project[]
               │  │   Layout: [a:bigint]
               │  └─ Project[]
               │     │   Layout: [a:bigint]
               │     └─ Values[]
               │            Layout: [a:bigint, row_number:bigint]

by the RemoveRedundantPredicateAboveTableScan rule, which is eliminating the intervening RemoteExchange. The plan fragmenter cannot currently deal with an empty Values node in the context of a partitioned subplan.

@martint martint mentioned this issue Nov 2, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
bug Something isn't working
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Allow empty values within partitioned fragment martint/trino
3 participants
@martint @findepi @phd3