Segfault on `analyse` in multinode setup. #3733

tanglebones · 2021-10-22T23:30:11Z

Relevant system information:

OS: docker desktop on windows 10
PostgreSQL version: PostgreSQL 13.3 on x86_64-pc-linux-musl, compiled by gcc (Alpine 10.2.1_pre1) 10.2.1 20201203, 64-bit
TimescaleDB version: 2.3.0
Installation method: using Docker

Describe the bug

I was following the instructions https://github.com/binakot/Multi-Node-TimescaleDB/tree/pgconf2021 to try out timescaledb and it is segfaulting.

To Reproduce

sync https://github.com/tanglebones/Multi-Node-TimescaleDB (I've forked the repo/branch to fix ports in the docker file.)
do:

docker network create pg_cluster_network
docker-compose up -d
gzip -k -d ./data/*csv.gz
docker exec -i pg_access_node /bin/sh < ./load-init-data.sh

via psql/etc to the access node: analyse telemetries;

Expected behavior

No crashing

Actual behavior

Segfault on data node:

2021-10-22 23:06:42.620 UTC [1] LOG:  server process (PID 240) was terminated by signal 11: Segmentation fault
2021-10-22 23:06:42.620 UTC [1] DETAIL:  Failed process was running: SELECT chunk_id,hypertable_id,att_num,nullfrac,width,distinctval,slotkind,slotopstrings,slotcollations,slot1numbers,slot2numbers,slot3numbers,slot4numbers,slot5numbers,slotvaluetypetrings,slot1values,slot2values,slot3values,slot4values,slot5values FROM _timescaledb_internal.get_chunk_colstats(relid => 'public.telemetries')

The text was updated successfully, but these errors were encountered:

nikkhils · 2021-10-25T14:03:33Z

@tanglebones will it be possible for you to try out the latest 2.4.2 release of timescaledb with Postgres 13.4?

nikkhils · 2021-10-25T15:26:12Z

@tanglebones the crash is occurring only when postgis extension is used in the table. I replaced the geography field type by text and subsequent ANALYZE does not crash.

nikkhils · 2021-10-26T05:45:53Z

The crash seems to be coming from the collect_colstat_slots function. It's kinda difficult to get any further information from the docker setup..

nikkhils · 2021-10-26T15:06:19Z

I managed to reproduced this locally. The stack trace is as below:

frame #3: 0x000000010c1b802d postgres`ExceptionalCondition(conditionName="HeapTupleIsValid(operator_tuple)", errorType="FailedAssertion", fileName="/Users/Nikhils/Work/TS/timescaledb/tsl/src/chunk_api.c", lineNumber=668) at assert.c:67:2
    frame #4: 0x000000010cadb9af timescaledb-tsl-2.5.0-dev.so`convert_op_oid_to_strings(op_id=0, result_strings=0x00007ffee41b9ad0) at chunk_api.c:668:2
  * frame #5: 0x000000010cadb4cb timescaledb-tsl-2.5.0-dev.so`collect_colstat_slots(tuple=0x00007faff8186cc8, formdata=0x00007faff8186d10, values=0x00007ffee41b9bf0, nulls=0x00007ffee41b9bd0) at chunk_api.c:767:3
    frame #6: 0x000000010cadb312 timescaledb-tsl-2.5.0-dev.so`chunk_get_single_colstats_tuple(chunk=0x00007fb001049938, column=5, tupdesc=0x00007fafc1016440) at chunk_api.c:909:2
    frame #7: 0x000000010cadad7b timescaledb-tsl-2.5.0-dev.so`chunk_api_fetch_next_colstats_tuple(funcctx=0x00007fb002831d20) at chunk_api.c:1465:11
    frame #8: 0x000000010cad8aa3 timescaledb-tsl-2.5.0-dev.so`chunk_api_get_chunk_stats(fcinfo=0x00007fb001030520, col_stats=true) at chunk_api.c:1640:22

The assert that is getting hit is:

Assert(HeapTupleIsValid(operator_tuple));

tanglebones · 2021-10-26T16:49:21Z

Tried with PostgreSQL: 13.4, PostGIS: 3.1.4, TimescaleDB: 2.4.2 and got the segfault. (Sorry, haven't had time to update the branch just yet with a repeatable test case). Removing PostGIS does prevent the segfault.

nikkhils · 2021-10-27T06:03:53Z

I am working on a fix for this and it will a part of the upcoming release.

Extensions like PostGIS introduce custom datatypes for columns. They also have custom statistics for such custom types. We already state in our comments that stats for custom types are not supported but it was never tested. Fix that now. Fixes timescale#3733

Extensions like PostGIS introduce custom datatypes for columns. They also have custom statistics for such custom types. We already state in our comments that stats for custom types are not supported but it was never tested. Fix that now. Fixes #3733

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * timescale#3034 Add support for PostgreSQL 14 * timescale#3435 Add continuous aggregates for distributed hypertables * timescale#3505 Add support for timezones in `time_bucket_ng()` * timescale#3598 Improve evaluation of stable functions such as now() on access node * timescale#3717 Support transparent decompression on individual chunks **Bugfixes** * timescale#3580 Fix memory context bug executing TRUNCATE * timescale#3592 Allow alter column type on distributed hypertable * timescale#3618 Fix execution of refresh_caggs from user actions * timescale#3625 Add shared dependencies when creating chunk * timescale#3626 Fix memory context bug executing TRUNCATE * timescale#3627 Schema qualify UDTs in multi-node * timescale#3638 Allow owner change of a data node * timescale#3654 Fix index attnum mapping in reorder_chunk * timescale#3661 Fix SkipScan path generation with constant DISTINCT column * timescale#3667 Fix compress_policy for multi txn handling * timescale#3673 Fix distributed hypertable DROP within a procedure * timescale#3701 Allow anyone to use size utilities on distributed hypertables * timescale#3708 Fix crash in get_aggsplit * timescale#3709 Fix ordered append pathkey check * timescale#3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * timescale#3724 Fix inserts into compressed chunks on hypertables with caggs * timescale#3727 Fix DirectFunctionCall crash in distributed_exec * timescale#3728 Fix SkipScan with varchar column * timescale#3733 Fix ANALYZE crash with custom statistics for custom types * timescale#3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * timescale#3034 Add support for PostgreSQL 14 * timescale#3435 Add continuous aggregates for distributed hypertables * timescale#3505 Add support for timezones in `time_bucket_ng()` **Bugfixes** * timescale#3580 Fix memory context bug executing TRUNCATE * timescale#3592 Allow alter column type on distributed hypertable * timescale#3598 Improve evaluation of stable functions such as now() on access node * timescale#3618 Fix execution of refresh_caggs from user actions * timescale#3625 Add shared dependencies when creating chunk * timescale#3626 Fix memory context bug executing TRUNCATE * timescale#3627 Schema qualify UDTs in multi-node * timescale#3638 Allow owner change of a data node * timescale#3654 Fix index attnum mapping in reorder_chunk * timescale#3661 Fix SkipScan path generation with constant DISTINCT column * timescale#3667 Fix compress_policy for multi txn handling * timescale#3673 Fix distributed hypertable DROP within a procedure * timescale#3701 Allow anyone to use size utilities on distributed hypertables * timescale#3708 Fix crash in get_aggsplit * timescale#3709 Fix ordered append pathkey check * timescale#3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * timescale#3717 Support transparent decompression on individual chunks * timescale#3724 Fix inserts into compressed chunks on hypertables with caggs * timescale#3727 Fix DirectFunctionCall crash in distributed_exec * timescale#3728 Fix SkipScan with varchar column * timescale#3733 Fix ANALYZE crash with custom statistics for custom types * timescale#3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * #3034 Add support for PostgreSQL 14 * #3435 Add continuous aggregates for distributed hypertables * #3505 Add support for timezones in `time_bucket_ng()` **Bugfixes** * #3580 Fix memory context bug executing TRUNCATE * #3592 Allow alter column type on distributed hypertable * #3598 Improve evaluation of stable functions such as now() on access node * #3618 Fix execution of refresh_caggs from user actions * #3625 Add shared dependencies when creating chunk * #3626 Fix memory context bug executing TRUNCATE * #3627 Schema qualify UDTs in multi-node * #3638 Allow owner change of a data node * #3654 Fix index attnum mapping in reorder_chunk * #3661 Fix SkipScan path generation with constant DISTINCT column * #3667 Fix compress_policy for multi txn handling * #3673 Fix distributed hypertable DROP within a procedure * #3701 Allow anyone to use size utilities on distributed hypertables * #3708 Fix crash in get_aggsplit * #3709 Fix ordered append pathkey check * #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * #3717 Support transparent decompression on individual chunks * #3724 Fix inserts into compressed chunks on hypertables with caggs * #3727 Fix DirectFunctionCall crash in distributed_exec * #3728 Fix SkipScan with varchar column * #3733 Fix ANALYZE crash with custom statistics for custom types * #3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * #3034 Add support for PostgreSQL 14 * #3435 Add continuous aggregates for distributed hypertables * #3505 Add support for timezones in `time_bucket_ng()` **Bugfixes** * #3580 Fix memory context bug executing TRUNCATE * #3592 Allow alter column type on distributed hypertable * #3598 Improve evaluation of stable functions such as now() on access node * #3618 Fix execution of refresh_caggs from user actions * #3625 Add shared dependencies when creating chunk * #3626 Fix memory context bug executing TRUNCATE * #3627 Schema qualify UDTs in multi-node * #3638 Allow owner change of a data node * #3654 Fix index attnum mapping in reorder_chunk * #3661 Fix SkipScan path generation with constant DISTINCT column * #3667 Fix compress_policy for multi txn handling * #3673 Fix distributed hypertable DROP within a procedure * #3701 Allow anyone to use size utilities on distributed hypertables * #3708 Fix crash in get_aggsplit * #3709 Fix ordered append pathkey check * #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * #3717 Support transparent decompression on individual chunks * #3724 Fix inserts into compressed chunks on hypertables with caggs * #3727 Fix DirectFunctionCall crash in distributed_exec * #3728 Fix SkipScan with varchar column * #3733 Fix ANALYZE crash with custom statistics for custom types * #3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * #3034 Add support for PostgreSQL 14 * #3435 Add continuous aggregates for distributed hypertables * #3505 Add support for timezones in `time_bucket_ng()` **Bugfixes** * #3580 Fix memory context bug executing TRUNCATE * #3592 Allow alter column type on distributed hypertable * #3598 Improve evaluation of stable functions such as now() on access node * #3618 Fix execution of refresh_caggs from user actions * #3625 Add shared dependencies when creating chunk * #3626 Fix memory context bug executing TRUNCATE * #3627 Schema qualify UDTs in multi-node * #3638 Allow owner change of a data node * #3654 Fix index attnum mapping in reorder_chunk * #3661 Fix SkipScan path generation with constant DISTINCT column * #3667 Fix compress_policy for multi txn handling * #3673 Fix distributed hypertable DROP within a procedure * #3701 Allow anyone to use size utilities on distributed hypertables * #3708 Fix crash in get_aggsplit * #3709 Fix ordered append pathkey check * #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * #3717 Support transparent decompression on individual chunks * #3724 Fix inserts into compressed chunks on hypertables with caggs * #3727 Fix DirectFunctionCall crash in distributed_exec * #3728 Fix SkipScan with varchar column * #3733 Fix ANALYZE crash with custom statistics for custom types * #3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

@binakot

This release adds major new features since the 2.4.2 release. We deem it moderate priority for upgrading. This release includes these noteworthy features: * Continuous Aggregates for Distributed Hypertables * Support for PostgreSQL 14 * Experimental: Support for timezones in `time_bucket_ng()`, including the `origin` argument This release also includes several bug fixes. **Features** * #3034 Add support for PostgreSQL 14 * #3435 Add continuous aggregates for distributed hypertables * #3505 Add support for timezones in `time_bucket_ng()` **Bugfixes** * #3580 Fix memory context bug executing TRUNCATE * #3592 Allow alter column type on distributed hypertable * #3598 Improve evaluation of stable functions such as now() on access node * #3618 Fix execution of refresh_caggs from user actions * #3625 Add shared dependencies when creating chunk * #3626 Fix memory context bug executing TRUNCATE * #3627 Schema qualify UDTs in multi-node * #3638 Allow owner change of a data node * #3654 Fix index attnum mapping in reorder_chunk * #3661 Fix SkipScan path generation with constant DISTINCT column * #3667 Fix compress_policy for multi txn handling * #3673 Fix distributed hypertable DROP within a procedure * #3701 Allow anyone to use size utilities on distributed hypertables * #3708 Fix crash in get_aggsplit * #3709 Fix ordered append pathkey check * #3712 Fix GRANT/REVOKE ALL IN SCHEMA handling * #3717 Support transparent decompression on individual chunks * #3724 Fix inserts into compressed chunks on hypertables with caggs * #3727 Fix DirectFunctionCall crash in distributed_exec * #3728 Fix SkipScan with varchar column * #3733 Fix ANALYZE crash with custom statistics for custom types * #3747 Always reset expr context in DecompressChunk **Thanks** * @binakot and @sebvett for reporting an issue with DISTINCT queries * @hardikm10, @DavidPavlicek and @pafiti for reporting bugs on TRUNCATE * @mjf for reporting an issue with ordered append and JOINs * @phemmer for reporting the issues on multinode with aggregate queries and evaluation of now() * @abolognino for reporting an issue with INSERTs into compressed hypertables that have cagg * @tanglebones for reporting the ANALYZE crash with custom types on multinode

tanglebones · 2021-10-28T16:24:36Z

@nikkhils Thanks for the quick replies and fix.

NunoFilipeSantos added multinode segfault Segmentation fault labels Oct 25, 2021

nikkhils self-assigned this Oct 27, 2021

nikkhils added the postgis label Oct 27, 2021

nikkhils mentioned this issue Oct 27, 2021

Fix ANALYZE crash with custom types #3745

Merged

nikkhils closed this as completed in #3745 Oct 27, 2021

fabriziomello mentioned this issue Oct 27, 2021

Release 2.5.0 #3744

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Segfault on `analyse` in multinode setup. #3733

Segfault on `analyse` in multinode setup. #3733

tanglebones commented Oct 22, 2021

nikkhils commented Oct 25, 2021

nikkhils commented Oct 25, 2021

nikkhils commented Oct 26, 2021

nikkhils commented Oct 26, 2021 •

edited

Loading

tanglebones commented Oct 26, 2021

nikkhils commented Oct 27, 2021

tanglebones commented Oct 28, 2021

Segfault on analyse in multinode setup. #3733

Segfault on analyse in multinode setup. #3733

Comments

tanglebones commented Oct 22, 2021

nikkhils commented Oct 25, 2021

nikkhils commented Oct 25, 2021

nikkhils commented Oct 26, 2021

nikkhils commented Oct 26, 2021 • edited Loading

tanglebones commented Oct 26, 2021

nikkhils commented Oct 27, 2021

tanglebones commented Oct 28, 2021

Segfault on `analyse` in multinode setup. #3733

Segfault on `analyse` in multinode setup. #3733

nikkhils commented Oct 26, 2021 •

edited

Loading