Add minmax sparse indexes when compressing columns with btree indexes #6705

akuzm · 2024-02-27T19:45:39Z

The decision to add a minmax sparse index is made every time when the compressed chunk is created (full decompression followed by compression) based on the currently present indexes on the hypertable. No new chunk compression settings are added.

No action is required on upgrade, but the feature is not enabled on existing chunks. The minmax index will be added when the chunk is fully decompressed and compressed.

No action is required on downgrade, we ignore the unknown metadata columns. They will be removed when the chunk is fully decompressed and compressed.

The potential drawback of this feature is increasing the storage requirements for the compressed chunk table, but it is normally only a few percent of the total compressed data size. It can be disabled with the GUC timescaledb.auto_sparse_indexes.

Here's a small example of this feature in action: https://gist.github.com/akuzm/84d4b3b609e3581768173bd21001dfbf
Note that the number of hit buffers is reduced almost 4x.

compressing

codecov · 2024-02-27T19:54:30Z

Codecov Report

Attention: Patch coverage is 84.61538% with 12 lines in your changes are missing coverage. Please review.

Project coverage is 80.98%. Comparing base (59f50f2) to head (ae5ec43).
Report is 67 commits behind head on main.

Files	Patch %	Lines
tsl/src/compression/create.c	84.41%	2 Missing and 10 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6705      +/-   ##
==========================================
+ Coverage   80.06%   80.98%   +0.91%     
==========================================
  Files         190      191       +1     
  Lines       37181    36406     -775     
  Branches     9450     9451       +1     
==========================================
- Hits        29770    29482     -288     
- Misses       2997     3163     +166     
+ Partials     4414     3761     -653

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

tsl/src/compression/create.c

fabriziomello · 2024-02-29T17:20:35Z

tsl/src/compression/create.c

+		IndexInfo *index_info = BuildIndexInfo(index_rel);
+		index_close(index_rel, NoLock);
+
+		if (index_info->ii_Am != BTREE_AM_OID)


Sorry for dumb question but what about partial indexes (aka index_info->ii_Predicate != NIL) ??

Not sure about what's the best approach in this case, but at least even if we add this metadata, the result should still be correct. Maybe we should create minmax all the columns from predicate as well.

Got it. Maybe adding predicate columns metadata can be part of a following PR.

Why are we tying this to btree?

Why are we tying this to btree?

The idea is that btree and sparse minmax index can be applied for similar types of queries.

If we have a gin index on a text column, the sparse minmax won't help the text search queries, so no point in creating it.

If we have a brin index, we have to look at the opclass -- if it's a bloom filter, minmax probably will be not so useful, but if it's a range brin, then minmax works. I haven't implemented this check yet.

svenklemm · 2024-03-03T09:00:03Z

tsl/src/compression/create.h

 #define COMPRESSION_COLUMN_METADATA_PATTERN_V1 "_ts_meta_%s_%d"

+/* Version 2 is indexed by the uncompressed chunk attno, not by the orderby index. */
+#define COMPRESSION_COLUMN_METADATA_PATTERN_V2 "_ts_meta_v2_%s_%d"


Should we use the column name here instead of thte attno?

dump/restore may change attno so i dont think its safe to use in this context

Changed to use the column names.

svenklemm

I dont think we can safely use attno as part of column name as that might change during dump/restore.

svenklemm · 2024-03-11T11:10:45Z

Have you thought about persisting this information in the compression settings in our catalog? We probably want to make this configurable at some point. So even if we dont allow direct modification now, having this information in our catalog would be a step in that direction.

akuzm · 2024-03-11T14:35:19Z

Have you thought about persisting this information in the compression settings in our catalog? We probably want to make this configurable at some point. So even if we dont allow direct modification now, having this information in our catalog would be a step in that direction.

For current functionality, the custom catalog info won't really help, I think it even complicates the things -- e.g. we have to think about how the implicit sparse index creation based on uncompressed btree is going to interact with the explicitly specified indexes. I'll probably have to implement fully configurable sparse indexes to understand this well. Do you think we should do it upfront, or this PR is usable enough on its own?

svenklemm · 2024-03-11T15:19:23Z

Have you thought about persisting this information in the compression settings in our catalog? We probably want to make this configurable at some point. So even if we dont allow direct modification now, having this information in our catalog would be a step in that direction.

For current functionality, the custom catalog info won't really help, I think it even complicates the things -- e.g. we have to think about how the implicit sparse index creation based on uncompressed btree is going to interact with the explicitly specified indexes. I'll probably have to implement fully configurable sparse indexes to understand this well. Do you think we should do it upfront, or this PR is usable enough on its own?

It will make this PR slightly more complex but I think we need this anyway. I think we should add a new table option for this similar to compress_segmentby but it doesnt have to be as part of this PR

PR timescale#6705 introduces the function is_sparse_index_type. However, it is only used for assert checking. If no assets are checked, the function is unused, and some compilers error out. This PR includes the function only if asserts are checked.

PR #6705 introduces the function is_sparse_index_type. However, it is only used for assert checking. If no assets are checked, the function is unused, and some compilers error out. This PR includes the function only if asserts are checked.

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [timescale#6786](timescale#6797). **Features** * timescale#6382 Support for time_bucket with origin and offset in CAggs * timescale#6696 Improve defaults for compression segment_by and order_by * timescale#6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * timescale#6754 Allow DROP CONSTRAINT on compressed hypertables * timescale#6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * timescale#6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * timescale#6810 Add telemetry for access methods * timescale#6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * timescale#6837 Add migration path for CAggs using time_bucket_ng * timescale#6865 Update the watermark when truncating a CAgg **Bugfixes** * timescale#6617 Fix error in show_chunks * timescale#6621 Remove metadata when dropping chunks * timescale#6677 Fix snapshot usage in CAgg invalidation scanner * timescale#6698 Define meaning of 0 retries for jobs as no retries * timescale#6717 Fix handling of compressed tables with primary or unique index in COPY path * timescale#6726 Fix constify cagg_watermark using window function when querying a CAgg * timescale#6729 Fix NULL start value handling in CAgg refresh * timescale#6732 Fix CAgg migration with custom timezone / date format settings * timescale#6752 Remove custom autovacuum setting from compressed chunks * timescale#6770 Fix plantime chunk exclusion for OSM chunk * timescale#6789 Fix deletes with subqueries and compression * timescale#6796 Fix a crash involving a view on a hypertable * timescale#6797 Fix foreign key constraint handling on compressed hypertables * timescale#6816 Fix handling of chunks with no contraints * timescale#6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * timescale#6849 Use non-orderby compressed metadata in compressed DML * timescale#6867 Clean up compression settings when deleting compressed cagg * timescale#6869 Fix compressed DML with constraints of form value OP column * timescale#6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

@brasic

This release contains performance improvements and bug fixes since the 2.14.2 release. We recommend that you upgrade at the next available opportunity. In addition, it includes these noteworthy features: * Support `time_bucket` with `origin` and/or `offset` on Continuous Aggregate * Compression improvements: - Improve expression pushdown - Add minmax sparse indexes when compressing columns with btree indexes - Make compression use the defaults functions - Vectorize filters in WHERE clause that contain text equality operators and LIKE expressions **Deprecation warning** * Starting on this release will not be possible to create Continuous Aggregate using `time_bucket_ng` anymore and it will be completely removed on the upcoming releases. * Recommend users to [migrate their old Continuous Aggregate format to the new one](https://docs.timescale.com/use-timescale/latest/continuous-aggregates/migrate/) because it support will be completely removed in next releases prevent them to migrate. * This is the last release supporting PostgreSQL 13. **For on-premise users and this release only**, you will need to run [this SQL script](https://github.com/timescale/timescaledb-extras/blob/master/utils/2.15.X-fix_hypertable_foreign_keys.sql) after running `ALTER EXTENSION`. More details can be found in the pull request [#6786](#6797). **Features** * #6382 Support for time_bucket with origin and offset in CAggs * #6696 Improve defaults for compression segment_by and order_by * #6705 Add sparse minmax indexes for compressed columns that have uncompressed btree indexes * #6754 Allow DROP CONSTRAINT on compressed hypertables * #6767 Add metadata table `_timestaledb_internal.bgw_job_stat_history` for tracking job execution history * #6798 Prevent usage of deprecated time_bucket_ng in CAgg definition * #6810 Add telemetry for access methods * #6811 Remove no longer relevant timescaledb.allow_install_without_preload GUC * #6837 Add migration path for CAggs using time_bucket_ng * #6865 Update the watermark when truncating a CAgg **Bugfixes** * #6617 Fix error in show_chunks * #6621 Remove metadata when dropping chunks * #6677 Fix snapshot usage in CAgg invalidation scanner * #6698 Define meaning of 0 retries for jobs as no retries * #6717 Fix handling of compressed tables with primary or unique index in COPY path * #6726 Fix constify cagg_watermark using window function when querying a CAgg * #6729 Fix NULL start value handling in CAgg refresh * #6732 Fix CAgg migration with custom timezone / date format settings * #6752 Remove custom autovacuum setting from compressed chunks * #6770 Fix plantime chunk exclusion for OSM chunk * #6789 Fix deletes with subqueries and compression * #6796 Fix a crash involving a view on a hypertable * #6797 Fix foreign key constraint handling on compressed hypertables * #6816 Fix handling of chunks with no contraints * #6820 Fix a crash when the ts_hypertable_insert_blocker was called directly * #6849 Use non-orderby compressed metadata in compressed DML * #6867 Clean up compression settings when deleting compressed cagg * #6869 Fix compressed DML with constraints of form value OP column * #6870 Fix bool expression pushdown for queries on compressed chunks **Thanks** * @brasic for reporting a crash when the ts_hypertable_insert_blocker was called directly * @bvanelli for reporting an issue with the jobs retry count * @djzurawsk For reporting error when dropping chunks * @Dzuzepppe for reporting an issue with DELETEs using subquery on compressed chunk working incorrectly. * @hongquan For reporting a 'timestamp out of range' error during CAgg migrations * @kevcenteno for reporting an issue with the show_chunks API showing incorrect output when 'created_before/created_after' was used with time-partitioned columns. * @mahipv For starting working on the job history PR * @rovo89 For reporting constify cagg_watermark not working using window function when querying a CAgg

akuzm added 2 commits February 27, 2024 22:43

For columns with btree indexes, create sparse minmax indexes when

be3959d

compressing

fix

da1a68d

add a test

9368da7

akuzm marked this pull request as ready for review February 29, 2024 10:15

akuzm self-assigned this Feb 29, 2024

fabriziomello reviewed Feb 29, 2024

View reviewed changes

tsl/src/compression/create.c Outdated Show resolved Hide resolved

fabriziomello reviewed Feb 29, 2024

View reviewed changes

fabriziomello approved these changes Feb 29, 2024

View reviewed changes

akuzm added 2 commits March 1, 2024 13:50

Merge remote-tracking branch 'origin/main' into HEAD

5076d86

review fixes

a79c8ae

svenklemm reviewed Mar 3, 2024

View reviewed changes

svenklemm requested changes Mar 3, 2024

View reviewed changes

akuzm added 11 commits March 4, 2024 12:42

use names instead of attnos

4d416a2

Merge remote-tracking branch 'origin/main' into HEAD

95ee24d

compat

112ec6c

compat

7ad8db0

compat

7311868

compat

f129c8f

comment

e6e9ec5

fix

094d123

codespell

aa4f76d

add a guc

2679193

support renames

fa2f5d0

svenklemm approved these changes Mar 11, 2024

View reviewed changes

akuzm added 2 commits March 12, 2024 13:03

Merge remote-tracking branch 'origin/main' into HEAD

75f842d

cleanup

ae5ec43

akuzm enabled auto-merge (squash) March 12, 2024 10:59

akuzm merged commit e306991 into timescale:main Mar 12, 2024
40 of 41 checks passed

akuzm deleted the btree-sparse branch March 12, 2024 10:59

akuzm mentioned this pull request Mar 12, 2024

Allow secondary indexes on compressed chunks #2418

Open

jnidzwetzki mentioned this pull request Mar 13, 2024

Exclude is_sparse_index_type in non assert builds #6764

Merged

fabriziomello mentioned this pull request Apr 30, 2024

Release notes header 2.15.0 #6874

Merged

fabriziomello mentioned this pull request May 6, 2024

Release 2.15.0 #6887

Merged

fabriziomello mentioned this pull request May 6, 2024

Release 2.15.0: Release branch #6889

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add minmax sparse indexes when compressing columns with btree indexes #6705

Add minmax sparse indexes when compressing columns with btree indexes #6705

akuzm commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading

fabriziomello Feb 29, 2024

akuzm Feb 29, 2024

fabriziomello Feb 29, 2024

svenklemm Mar 3, 2024

akuzm Mar 4, 2024

svenklemm Mar 3, 2024

svenklemm Mar 3, 2024

akuzm Mar 4, 2024

svenklemm left a comment

svenklemm commented Mar 11, 2024

akuzm commented Mar 11, 2024

svenklemm commented Mar 11, 2024

Add minmax sparse indexes when compressing columns with btree indexes #6705

Add minmax sparse indexes when compressing columns with btree indexes #6705

Conversation

akuzm commented Feb 27, 2024 • edited Loading

codecov bot commented Feb 27, 2024 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

svenklemm left a comment

Choose a reason for hiding this comment

svenklemm commented Mar 11, 2024

akuzm commented Mar 11, 2024

svenklemm commented Mar 11, 2024

akuzm commented Feb 27, 2024 •

edited

Loading

codecov bot commented Feb 27, 2024 •

edited

Loading