-
Notifications
You must be signed in to change notification settings - Fork 875
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Very high planning time for queries on real-time continous aggregates with many chunks #6105
Comments
Hello @raymalt, Thanks for the detailed steps to reproduce the issue. I was able to reproduce it locally using TimescaleDB 2.12.0-dev. The hypertable Flamegraphsmaterialized_only = truematerialized_only = falseDiff |
Hi @jnidzwetzki, Thanks for your response. It seems like the WHERE-clause is not propagated when excluding chunks in the raw table. For example, these queries on the raw table have about the same planning times as for the queries on the CAGGs above. Query 1,
|
Hello, I just wanted to add that this problem is very dependent on the use of the
|
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: #6105, #6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: #6105, #6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
What type of bug is this?
Performance issue
What subsystems and features are affected?
Query planner
What happened?
A simple query on a continuous aggregate extracting the last timestamp takes unexpectedly long time due to the time spent planning the query. It seems to be very dependent on the number of chunks in the raw table. Setting
materialized_only = true
however gives a super-fast response.It is expected to take more time on a real-time CAGG than if only working on the materialized view, but not to spend this much time query planning - especially considering that the query filters out all data in the WHERE clause.
Query in reproduction steps below (Table creation courtesy of @jnidzwetzki) ;)
TimescaleDB version affected
2.11.2
PostgreSQL version used
15.4
What operating system did you use?
Timescale Cloud
What installation method did you use?
Not applicable
What platform did you run on?
Timescale Cloud
Relevant log output and stack trace
With
materialized_only = true
The text was updated successfully, but these errors were encountered: