-
Notifications
You must be signed in to change notification settings - Fork 869
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[Bug]: Catastrophic planning times for Realtime Continuous Aggregates (issues with _timescaledb_functions.cagg_watermark
)
#6321
Comments
_timescaledb_functions.cagg_watermark
)
A few more bits of info that might be useful:
|
Hello @martinhale, Thanks for reaching out and making us aware of these high planning times. I was able to reproduce the problem in my local environment using TimescaleDB 2.13.0-dev and PostgreSQL 15. For reproduction, I used a hypertable with 8000 chunks: CREATE TABLE ht_chunk_8k(time timestamptz, device int, value float);
SELECT create_hypertable('ht_chunk_8k','time',chunk_time_interval:='1d'::interval);
INSERT INTO ht_chunk_8k SELECT '1900-01-01'::timestamptz + (interval '1 day' * i), 1, 0.5 FROM generate_series(1,8000) g(i);
CREATE MATERIALIZED VIEW continuous_aggregate_hourly
WITH (timescaledb.continuous) AS SELECT time_bucket('1 hour', time) AS bucket, device, avg(value) FROM ht_chunk_8k GROUP BY bucket, device;
ALTER MATERIALIZED VIEW continuous_aggregate_hourly set (timescaledb.materialized_only = false);
CALL refresh_continuous_aggregate('continuous_aggregate_hourly', '1900-01-01', '2021-06-01'); After activating the real-time functionality of the continuous aggregate, I also observed high planning times when the continuous aggregate is accessed. |
#6105 seems to be related |
Hey @jnidzwetzki - thanks so much for reproducing.
Yeah, I thought so too, although I wanted to ensure |
It will be interesting to see if this issue can be traced to a change made in a recent Timescale update or whether it was present all along. I'm sure I've seen a performance reduction in certain Grafana visualisations targetting real-time CAGGs but it wasn't so bad that it made me look into it further. I replicated the queries at the top of the issue on a number of our CAGGs and saw planning time increases of around x20. This is nowhere near the x1000 example given but our CAGGs have far fewer chunks. |
Hey @jnidzwetzki! I noticed that you've got a PR open that addresses this issue (with some 🌶️ benchmark number improvements - congratulations!) Any idea when we might be able to expect the fix to be merged and hit a release? Many thanks! |
Hello @martinhale, The PR is currently in an early prototype state. But it shows very promising numbers. We are currently testing the correctness of the change. At the moment, I cannot say anything definite about merging these changes. However, if the test goes smoothly, we will soon work on a production-ready version of these changes. |
@jnidzwetzki @fabriziomello - I don't suppose you have any further updates on this at the moment? It's causing us quite a lot of issues in our production environments and we're trying to decide how much effort we should put into mitigation ourselves vs waiting for the fix. Thanks! 🙂 |
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
Hello @martinhale, The PR is currently under review and we plan to include this improvement in TimescaleDB 2.14.0. |
Amazing, thanks so much for the update! And thanks for the huge effort involved in implementing the fix - I appreciate this hasn't been a trivial thing to resolve! |
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: #6105, #6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: timescale#6105, timescale#6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
The watermark function for CAggs is declared as STABLE since the value of the function changes after every CAgg refresh. The function volatility prevents the planner from replacing the function invocation with a constant value and executing plan time chunk exclusion. This leads to high planning times on hypertables with many chunks. This PR replaces the function invocation with a constant value to allow plan time exclusion of chunks. We perform the replacement at plan time instead of changing the function volatility to IMMUTABLE, because we want to control the constification. Only queries that access the underlying hypertable in a query (i.e., no queries like SELECT cagg_watermark(...) without any FROM condition) are rewritten. This is done to make sure that the query is properly invalidated when the underlying table changes (e.g., the watermark is updated) and the query is replanned on the subsequent execution. Fixes: #6105, #6321 Co-authored-by: Fabrizio de Royes Mello <fabriziomello@gmail.com>
What type of bug is this?
Performance issue
What subsystems and features are affected?
Continuous aggregate, Query planner
What happened?
We're facing incredibly slow planning times when querying realtime continuous aggregates over large periods of time.
Here’s an example query and EXPLAIN against a CAgg:
And the same query / EXPLAIN against the underlying hypertable:
The execution times are slightly different (which might be expected as these are one-off queries etc), but the planning differences are vast (and repeatable).
28,101
ms vs27 ms
(1000x slower!) Note - in the above example thecagg_watermark
is later than2023-11-13 23:59:59
so the result sets are identical.To dig a little deeper, I recreated the realtime functionality with a UNION between the underlying CAgg hypertable and the raw hypertable myself, like this:
This also has the terrible planning performance.
However, if I replace the
_timescaledb_functions.to_timestamp(_timescaledb_functions.cagg_watermark(166))
call with its result (acquired in a previous statement), then the query absolutely flies:So, it seems like the calls to the watermark function are not being constified (or something along those lines).
Are we doing something wrong? Is there something we can do to improve this performance beyond abandoning realtime aggregates / rolling our own?
Many thanks!
Slack thread with similar info.
TimescaleDB version affected
2.12.1
PostgreSQL version used
13.13
What operating system did you use?
Managed Service for Timescale (Aiven)
What installation method did you use?
Not applicable
What platform did you run on?
Managed Service for TimescaleDB (MST/Aiven)
Relevant log output and stack trace
No response
How can we reproduce the bug?
The text was updated successfully, but these errors were encountered: