Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Do not modify aggregation state in finalize #4219

Merged
merged 1 commit into from
Apr 6, 2022

Conversation

mkindahl
Copy link
Contributor

@mkindahl mkindahl commented Apr 6, 2022

The function tsl_finalize_agg_ffunc modified the aggregation state by
setting trans_value to the final result when computing the final
value. Since the state can be re-used several times, there could be
several calls to the finalization function, and the finalization
function would be confused when passed a final value instead of a
aggregation state transition value.

This commit fixes this by not modifying the trans_value when
computing the final value and instead just returns it (or the original
trans_value if there is no finalization function).

Fixes #3248

@mkindahl mkindahl self-assigned this Apr 6, 2022
@mfundul mfundul self-requested a review April 6, 2022 15:56
Comment on lines +524 to +526
* The state passed down to this function can be used by other aggregate
* functions so it is important to not change the state when computing the
* final result.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't know about this. Is it some optimization like using the same state for avg, count and sum?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the comment should say something in the spirit of "the partial state should not be modified by the finalization function because it can be reused". I thought the same thing about avg and sum but apparently the reproducer only has 1 aggregate function.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Didn't know about this. Is it some optimization like using the same state for avg, count and sum?

It seems to be mainly used in hash aggregation, but I see it pop up in group aggregation as well. It is the finalize_aggregates function in nodeAgg.c that is calling this using the hashed aggregates.

@mkindahl mkindahl force-pushed the fix-finalize-partials branch from 96d893c to 9412bb4 Compare April 6, 2022 17:44
@mkindahl mkindahl marked this pull request as ready for review April 6, 2022 17:44
@mkindahl mkindahl requested a review from a team as a code owner April 6, 2022 17:44
@mkindahl mkindahl requested review from josesahad and removed request for a team April 6, 2022 17:44
The function `tsl_finalize_agg_ffunc` modified the aggregation state by
setting `trans_value` to the final result when computing the final
value. Since the state can be re-used several times, there could be
several calls to the finalization function, and the finalization
function would be confused when passed a final value instead of a
aggregation state transition value.

This commit fixes this by not modifying the `trans_value` when
computing the final value and instead just returns it (or the original
`trans_value` if there is no finalization function).

Fixes timescale#3248
@mkindahl mkindahl force-pushed the fix-finalize-partials branch from 9412bb4 to 3ad59bc Compare April 6, 2022 17:45
@codecov
Copy link

codecov bot commented Apr 6, 2022

Codecov Report

Merging #4219 (3ad59bc) into main (a064fd3) will increase coverage by 0.00%.
The diff coverage is 100.00%.

Impacted file tree graph

@@           Coverage Diff           @@
##             main    #4219   +/-   ##
=======================================
  Coverage   90.78%   90.79%           
=======================================
  Files         215      215           
  Lines       39530    39537    +7     
=======================================
+ Hits        35888    35898   +10     
+ Misses       3642     3639    -3     
Impacted Files Coverage Δ
tsl/src/fdw/data_node_scan_exec.c 88.88% <ø> (-0.40%) ⬇️
src/nodes/hypertable_modify.c 70.10% <100.00%> (+0.51%) ⬆️
src/planner.c 95.19% <100.00%> (+0.01%) ⬆️
tsl/src/fdw/data_node_scan_plan.c 98.23% <100.00%> (+0.01%) ⬆️
tsl/src/partialize_finalize.c 96.24% <100.00%> (+0.03%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update ae50a53...3ad59bc. Read the comment docs.

@mkindahl
Copy link
Contributor Author

mkindahl commented Apr 6, 2022

Using the example from #3248 this is the result:

mats=# \ir t3248.sql
CREATE TABLE
CREATE INDEX
CREATE INDEX
   create_hypertable    
------------------------
 (265,public,metrics,t)
(1 row)

ALTER TABLE
INSERT 0 75
ALTER TABLE
INSERT 0 105
ALTER TABLE
INSERT 0 105
ANALYZE
 count 
-------
   285
(1 row)

psql:t3248.sql:25: NOTICE:  refreshing continuous aggregate "cagg1"
HINT:  Use WITH NO DATA if you do not want to refresh the continuous aggregate on creation.
CREATE MATERIALIZED VIEW
psql:t3248.sql:27: server closed the connection unexpectedly
        This probably means the server terminated abnormally
        before or while processing the request.
psql:t3248.sql:27: fatal: connection to server was lost

@mkindahl mkindahl merged commit 1b2926c into timescale:main Apr 6, 2022
@mkindahl mkindahl deleted the fix-finalize-partials branch April 6, 2022 18:50
@mkindahl mkindahl mentioned this pull request Apr 6, 2022
@mkindahl mkindahl added this to the TimescaleDB 2.6.1 milestone Apr 7, 2022
RafiaSabih added a commit to RafiaSabih/timescaledb that referenced this pull request Apr 8, 2022
This release is patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* timescale#4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* timescale#4122 Fix segfault on INSERT into distributed hypertable
* timescale#4142 Ignore invalid relid when deleting hypertable
* timescale#4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* timescale#4161 Fix memory handling during scans
* timescale#4176 Fix remote EXPLAIN with parameterized queries
* timescale#4181 Fix spelling errors and omissions
* timescale#4186 Fix owner change for distributed hypertable
* timescale#4192 Abort sessions after extension reload
* timescale#4193 Fix relcache callback handling causing crashes
* timescale#4199 Remove signal-unsafe calls from signal handlers
* timescale#4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
RafiaSabih added a commit to RafiaSabih/timescaledb that referenced this pull request Apr 11, 2022
This release is patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* timescale#4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* timescale#4122 Fix segfault on INSERT into distributed hypertable
* timescale#4142 Ignore invalid relid when deleting hypertable
* timescale#4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* timescale#4161 Fix memory handling during scans
* timescale#4176 Fix remote EXPLAIN with parameterized queries
* timescale#4181 Fix spelling errors and omissions
* timescale#4186 Fix owner change for distributed hypertable
* timescale#4192 Abort sessions after extension reload
* timescale#4193 Fix relcache callback handling causing crashes
* timescale#4199 Remove signal-unsafe calls from signal handlers
* timescale#4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
RafiaSabih added a commit to RafiaSabih/timescaledb that referenced this pull request Apr 11, 2022
This release is patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* timescale#4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* timescale#4122 Fix segfault on INSERT into distributed hypertable
* timescale#4142 Ignore invalid relid when deleting hypertable
* timescale#4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* timescale#4161 Fix memory handling during scans
* timescale#4176 Fix remote EXPLAIN with parameterized queries
* timescale#4181 Fix spelling errors and omissions
* timescale#4186 Fix owner change for distributed hypertable
* timescale#4192 Abort sessions after extension reload
* timescale#4193 Fix relcache callback handling causing crashes
* timescale#4199 Remove signal-unsafe calls from signal handlers
* timescale#4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
RafiaSabih added a commit to RafiaSabih/timescaledb that referenced this pull request Apr 11, 2022
This release is patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* timescale#4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* timescale#4122 Fix segfault on INSERT into distributed hypertable
* timescale#4142 Ignore invalid relid when deleting hypertable
* timescale#4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* timescale#4161 Fix memory handling during scans
* timescale#4176 Fix remote EXPLAIN with parameterized queries
* timescale#4181 Fix spelling errors and omissions
* timescale#4186 Fix owner change for distributed hypertable
* timescale#4192 Abort sessions after extension reload
* timescale#4193 Fix relcache callback handling causing crashes
* timescale#4199 Remove signal-unsafe calls from signal handlers
* timescale#4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
svenklemm pushed a commit to RafiaSabih/timescaledb that referenced this pull request Apr 11, 2022
This release is a patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* timescale#4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* timescale#4122 Fix segfault on INSERT into distributed hypertable
* timescale#4142 Ignore invalid relid when deleting hypertable
* timescale#4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* timescale#4161 Fix memory handling during scans
* timescale#4176 Fix remote EXPLAIN with parameterized queries
* timescale#4181 Fix spelling errors and omissions
* timescale#4186 Fix owner change for distributed hypertable
* timescale#4192 Abort sessions after extension reload
* timescale#4193 Fix relcache callback handling causing crashes
* timescale#4199 Remove signal-unsafe calls from signal handlers
* timescale#4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
svenklemm pushed a commit that referenced this pull request Apr 11, 2022
This release is a patch release. We recommend that you upgrade at the next available opportunity.

**Bugfixes**
* #4121 Fix RENAME TO/SET SCHEMA on distributed hypertable
* #4122 Fix segfault on INSERT into distributed hypertable
* #4142 Ignore invalid relid when deleting hypertable
* #4159 Fix ADD COLUMN IF NOT EXISTS error on compressed hypertable
* #4161 Fix memory handling during scans
* #4176 Fix remote EXPLAIN with parameterized queries
* #4181 Fix spelling errors and omissions
* #4186 Fix owner change for distributed hypertable
* #4192 Abort sessions after extension reload
* #4193 Fix relcache callback handling causing crashes
* #4199 Remove signal-unsafe calls from signal handlers
* #4219 Do not modify aggregation state in finalize

**Thanks**
* @abrownsword for reporting a crash in the telemetry reporter
* @daydayup863 for reporting issue with remote explain
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Segfault when querying continuous aggregate
4 participants