Fix delete on tables involving hypertables with compression #2092

svenklemm · 2020-07-16T21:17:35Z

The DML blocker to block INSERTs and UPDATEs on compressed hypertables
would trigger if the UPDATE or DELETE referenced any hypertable with
compressed chunks. This patch changes the logic to only block if the
target of the UPDATE or DELETE is a compressed hypertable.

Fixes #2091

mfreed · 2020-07-16T21:21:54Z

@svenklemm Quick clarification: Should the commit message say:

target of the UPDATE or DELETE is a compressed CHUNK WITHIN A hypertable

(Don't mean to be pedantic, just wasn't sure if this is exactly what you meant, i.e., does it only block if it would other DELETE/UPDATE a row that falls in a chunk currently compressed, or could it block under different scenarios as well.)

svenklemm · 2020-07-16T21:28:24Z

@svenklemm Quick clarification: Should the commit message say:
target of the UPDATE or DELETE is a compressed CHUNK WITHIN A hypertable
(Don't mean to be pedantic, just wasn't sure if this is exactly what you meant, i.e., does it only block if it would other DELETE/UPDATE a row that falls in a chunk currently compressed, or could it block under different scenarios as well.)

This part of the logic is not changed, so your suggested change is correct.

codecov · 2020-07-16T21:36:41Z

Codecov Report

Merging #2092 into master will increase coverage by 0.20%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #2092      +/-   ##
==========================================
+ Coverage   90.08%   90.29%   +0.20%     
==========================================
  Files         211      211              
  Lines       33647    33996     +349     
==========================================
+ Hits        30312    30697     +385     
+ Misses       3335     3299      -36

Impacted Files	Coverage Δ
src/planner.c	`94.25% <100.00%> (+0.05%)`	⬆️
tsl/src/nodes/decompress_chunk/exec.c	`96.36% <100.00%> (ø)`
src/loader/bgw_message_queue.c	`84.51% <0.00%> (-3.23%)`	⬇️
src/process_utility.c	`94.39% <0.00%> (+0.90%)`	⬆️
src/import/planner.c	`70.30% <0.00%> (+11.12%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 13aa729...112c566. Read the comment docs.

cevian

Thanks @svenklemm for the quick turnaround

cevian · 2020-07-16T21:37:52Z

tsl/src/nodes/decompress_chunk/exec.c

-			elog(ERROR, "transparent decompression only supports tableoid system column");
+		if (var->varattno < SelfItemPointerAttributeNumber)
+			elog(ERROR,
+				 "transparent decompression only supports tableoid system column %d",


this message should probably be reverted or made more explcit that the %d attno is not supported

cevian · 2020-07-16T21:51:14Z

src/planner.c

@@ -726,6 +726,18 @@ valid_hook_call(void)
 	return ts_extension_is_loaded() && planner_hcache_exists();
 }

+static bool
+dml_involves_hypertable(PlannerInfo *root, Hypertable *ht, Index rti)


does this still throw an error if the hypertable delete is inside a CTE. ie WITH cte as (DELETE FROM compressed_ht RETURNING *) SELECT * FROM CTE I ask because we get the resultRelation from the root and I'm not sure if thats the root of the query or the cte. In any case I don't think we have a test for that, but should.

root will be the root of the cte when planning the subquery, i've added a test

The DML blocker to block INSERTs and UPDATEs on compressed hypertables would trigger if the UPDATE or DELETE referenced any hypertable with compressed chunks. This patch changes the logic to only block if the target of the UPDATE or DELETE is a compressed chunk.

This patch enhances the license check script to check for a license header in isolation test files

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2150 Lock dimension slice tuple when scanning * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix negative bitmapset member not allowed in decompression **Thanks** * @akamensky for reporting an issue with drop_chunks * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2150 Lock dimension slice tuple when scanning * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2256 Fix segfault in chunk_append with space partitioning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2150 Lock dimension slice tuple when scanning * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2256 Fix segfault in chunk_append with space partitioning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2116 Propagate privileges from hypertables to chunks * timescale#2150 Lock dimension slice tuple when scanning * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2256 Fix segfault in chunk_append with space partitioning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2255 Propagate privileges from hypertables to chunks * timescale#2256 Fix segfault in chunk_append with space partitioning * timescale#2259 Fix recursion in cache processing * timescale#2261 Lock dimension slice tuple when scanning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @fvannee for reporting an issue with cache invalidation * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address bugs in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2255 Propagate privileges from hypertables to chunks * timescale#2256 Fix segfault in chunk_append with space partitioning * timescale#2259 Fix recursion in cache processing * timescale#2261 Lock dimension slice tuple when scanning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @fvannee for reporting an issue with cache invalidation * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in compression, drop_chunks and the background worker scheduler. **Bugfixes** * timescale#2059 Improve infering start and stop arguments from gapfill query * timescale#2067 Support moving compressed chunks * timescale#2068 Apply SET TABLESPACE for compressed chunks * timescale#2090 Fix index creation with IF NOT EXISTS for existing indexes * timescale#2092 Fix delete on tables involving hypertables with compression * timescale#2164 Fix telemetry installed_time format * timescale#2184 Fix background worker scheduler memory consumption * timescale#2222 Fix `negative bitmapset member not allowed` in decompression * timescale#2255 Propagate privileges from hypertables to chunks * timescale#2256 Fix segfault in chunk_append with space partitioning * timescale#2259 Fix recursion in cache processing * timescale#2261 Lock dimension slice tuple when scanning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @fvannee for reporting an issue with cache invalidation * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @prathamesh-sonpatki for contributing a typo fix * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in compression, drop_chunks and the background worker scheduler. **Bugfixes** * #2059 Improve infering start and stop arguments from gapfill query * #2067 Support moving compressed chunks * #2068 Apply SET TABLESPACE for compressed chunks * #2090 Fix index creation with IF NOT EXISTS for existing indexes * #2092 Fix delete on tables involving hypertables with compression * #2164 Fix telemetry installed_time format * #2184 Fix background worker scheduler memory consumption * #2222 Fix `negative bitmapset member not allowed` in decompression * #2255 Propagate privileges from hypertables to chunks * #2256 Fix segfault in chunk_append with space partitioning * #2259 Fix recursion in cache processing * #2261 Lock dimension slice tuple when scanning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @fvannee for reporting an issue with cache invalidation * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @prathamesh-sonpatki for contributing a typo fix * @sezaru for reporting an issue with background worker scheduler memory consumption

@akamensky

This maintenance release contains bugfixes since the 1.7.2 release. We deem it high priority for upgrading. In particular the fixes contained in this maintenance release address issues in compression, drop_chunks and the background worker scheduler. **Bugfixes** * #2059 Improve infering start and stop arguments from gapfill query * #2067 Support moving compressed chunks * #2068 Apply SET TABLESPACE for compressed chunks * #2090 Fix index creation with IF NOT EXISTS for existing indexes * #2092 Fix delete on tables involving hypertables with compression * #2164 Fix telemetry installed_time format * #2184 Fix background worker scheduler memory consumption * #2222 Fix `negative bitmapset member not allowed` in decompression * #2255 Propagate privileges from hypertables to chunks * #2256 Fix segfault in chunk_append with space partitioning * #2259 Fix recursion in cache processing * #2261 Lock dimension slice tuple when scanning **Thanks** * @akamensky for reporting an issue with drop_chunks and ChunkAppend with space partitioning * @dewetburger430 for reporting an issue with setting tablespace for compressed chunks * @fvannee for reporting an issue with cache invalidation * @nexces for reporting an issue with ChunkAppend on space-partitioned hypertables * @PichetGoulu for reporting an issue with index creation and IF NOT EXISTS * @prathamesh-sonpatki for contributing a typo fix * @sezaru for reporting an issue with background worker scheduler memory consumption

svenklemm requested a review from a team as a code owner July 16, 2020 21:17

svenklemm requested review from mkindahl, WireBaron and gayyappan and removed request for a team July 16, 2020 21:17

svenklemm force-pushed the compress_delete branch from 41360e3 to f93f87c Compare July 16, 2020 21:23

svenklemm requested review from cevian, erimatnor, k-rus and pmwkaa July 16, 2020 21:30

cevian reviewed Jul 16, 2020

View reviewed changes

svenklemm force-pushed the compress_delete branch 4 times, most recently from 9a5af14 to ae90e47 Compare July 17, 2020 01:14

gayyappan approved these changes Jul 17, 2020

View reviewed changes

cevian approved these changes Jul 17, 2020

View reviewed changes

svenklemm force-pushed the compress_delete branch from ae90e47 to 2696ab7 Compare July 20, 2020 08:42

svenklemm added 3 commits July 20, 2020 12:34

Add missing license header to spec files

b0ea27a

Add isolation test files to license check

112c566

This patch enhances the license check script to check for a license header in isolation test files

svenklemm force-pushed the compress_delete branch from 2696ab7 to 112c566 Compare July 20, 2020 10:40

svenklemm merged commit 5c70dfc into timescale:master Jul 20, 2020

svenklemm mentioned this pull request Aug 24, 2020

Release 1.7.3 #2254

Merged

cevian mentioned this pull request Feb 1, 2021

prom_api.execute_maintenance - cannot update/delete rows from chunk "_hyper_29_146874_chunk" as it is compressed timescale/promscale#454

Closed

svenklemm deleted the compress_delete branch April 18, 2021 14:26

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix delete on tables involving hypertables with compression #2092

Fix delete on tables involving hypertables with compression #2092

svenklemm commented Jul 16, 2020 •

edited

Loading

mfreed commented Jul 16, 2020

svenklemm commented Jul 16, 2020

codecov bot commented Jul 16, 2020 •

edited

Loading

cevian left a comment

cevian Jul 16, 2020

cevian Jul 16, 2020

svenklemm Jul 16, 2020

Fix delete on tables involving hypertables with compression #2092

Fix delete on tables involving hypertables with compression #2092

Conversation

svenklemm commented Jul 16, 2020 • edited Loading

mfreed commented Jul 16, 2020

svenklemm commented Jul 16, 2020

codecov bot commented Jul 16, 2020 • edited Loading

Codecov Report

cevian left a comment

Choose a reason for hiding this comment

cevian Jul 16, 2020

Choose a reason for hiding this comment

cevian Jul 16, 2020

Choose a reason for hiding this comment

svenklemm Jul 16, 2020

Choose a reason for hiding this comment

svenklemm commented Jul 16, 2020 •

edited

Loading

codecov bot commented Jul 16, 2020 •

edited

Loading