Segmentation fault,"Failed process was running: insert hypertable"

[Specter-Y]

Relevant system information:

• OS: [SUSE 12]
• PostgreSQL version (output of postgres --version): 11.2
• TimescaleDB version (output of \dx in psql):1.2.2
• Installation method: "source"

Describe the bug
error log:
2019-04-12 13:00:21.795 UTC,,,284641,,5cb037a2.457e1,6,,2019-04-12 07:00:50 UTC,,0,LOG,00000,"server process (PID 253718) was terminated by signal 11: Segmentation fault","Failed process was running: insert into t_mpp_11120_1d_rx0l3p_0(time,entity_id,ind_030ix5z4n,ind_03vfryh7a,ind_04apng40b,ind_05aa4w885,ind_05n4v6dpm,ind_05qpaty18,ind_05rkbs2ri,ind_09rvbyr2y,ind_0au4x42mk,ind_0b4g17rpg,ind_0cibtjd1m) select U0.time as time,U0.entity_id as entity_id,U0.ind_030ix5z4n as ind_030ix5z4n,U0.ind_03vfryh7a as ind_03vfryh7a,U0.ind_04apng40b as ind_04apng40b,U0.ind_05aa4w885 as ind_05aa4w885,U0.ind_05n4v6dpm as ind_05n4v6dpm,U0.ind_05qpaty18 as ind_05qpaty18,U0.ind_05rkbs2ri as ind_05rkbs2ri,U0.ind_09rvbyr2y as ind_09rvbyr2y,U0.ind_0au4x42mk as ind_0au4x42mk,U0.ind_0b4g17rpg as ind_0b4g17rpg,U0.ind_0cibtjd1m as ind_0cibtjd1m from (select t01.destination_instance as entity_id,dw_common.time_floor(t00.time,'30m') as time,avg(t00.ind_030ix5z4n) as ind_030ix5z4n,avg(t00.ind_03vfryh7a) as ind_03vfryh7a,avg(t00.ind_04apng40b) as ind_04apng40b,avg(t00.ind_05aa4w885) as ind_05aa4w885,avg(t00.ind_05n4v6dpm) as ind_05n4v6dpm,avg(t00.ind_05qpaty18) as ind_05qpaty18,avg(t00.ind_05rkbs2ri) as ind_05rkbs2ri,avg(t00.ind_09rvby",,,,,,,,""

coredump with gdb:
(gdb) where
#0 0x00007fec434cbd90 in ts_chunk_insert_state_create () from /opt/tsdb/pgsql/lib/timescaledb-1.2.2.so
#1 0x00007fec434c951f in ts_chunk_dispatch_get_chunk_insert_state () from /opt/tsdb/pgsql/lib/timescaledb-1.2.2.so
#2 0x00007fec434c98a4 in chunk_dispatch_exec () from /opt/tsdb/pgsql/lib/timescaledb-1.2.2.so
#3 0x000000000062704b in ExecModifyTable ()
#4 0x0000000000604daa in standard_ExecutorRun ()
#5 0x00007feec385a2f5 in pgss_ExecutorRun () from /opt/tsdb/pgsql/lib/pg_stat_statements.so
#6 0x000000000073cada in ProcessQuery ()
#7 0x000000000073cd01 in PortalRunMulti ()
#8 0x000000000073d7bc in PortalRun ()
#9 0x000000000073b2cd in PostgresMain ()
#10 0x000000000048e399 in ServerLoop ()
#11 0x00000000006d1eda in PostmasterMain ()
#12 0x000000000048edef in main ()

To Reproduce
USE TimescaleDB create hypertable , then in postgres the table like this:

CREATE TABLE dw_ies.t_mpp_11182_1d_ejki0_0
(
"time" timestamp with time zone NOT NULL,
entity_id text COLLATE pg_catalog."default" NOT NULL,
ind_042lgvatg numeric(25,5),
CONSTRAINT t_mpp_11182_1d_ejki0_0_pkey PRIMARY KEY ("time", entity_id)
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;

ALTER TABLE dw_ies.t_mpp_11182_1d_ejki0_0
OWNER to dw_ies;

CREATE INDEX t_mpp_11182_1d_ejki0_0_time_idx
ON dw_ies.t_mpp_11182_1d_ejki0_0 USING btree
("time" DESC)
TABLESPACE pg_default;

CREATE TRIGGER ts_insert_blocker
BEFORE INSERT
ON dw_ies.t_mpp_11182_1d_ejki0_0
FOR EACH ROW
EXECUTE PROCEDURE _timescaledb_internal.insert_blocker();

When a large amount of data is inserted and chunks are automatically created, the database sometimes crashes.

[cevian]

@Specter-Y any chance you still have the core dump around? Can you do a list command on it? I'd like to see what line of ts_chunk_insert_state_create the crash is on?

[Specter-Y]

It takes a long time to obtain the coredump with the code line. We use the -g parameter to recompile the timescaledb and postgres and install the environment again. The problem recurs and The key core dump information is as follows:
(gdb) bt
#0 chunk_insert_state_set_arbiter_indexes (state=0x1512aa0, state=0x1512aa0, dispatch=0x15114a0, chunk_rel=0x7fbf6826c818)
at /opt/software/timescaledb-1.2.2/src/chunk_insert_state.c:470
#1 ts_chunk_insert_state_create (chunk=chunk@entry=0x15dd8b0, dispatch=dispatch@entry=0x1605b40) at /opt/software/timescaledb-1.2.2/src/chunk_insert_state.c:550
#2 0x00007fbf68cbbcaf in ts_chunk_dispatch_get_chunk_insert_state (dispatch=dispatch@entry=0x1605b40, point=)
at /opt/software/timescaledb-1.2.2/src/chunk_dispatch.c:68
#3 0x00007fbf68cbc054 in chunk_dispatch_exec (node=0x13df280) at /opt/software/timescaledb-1.2.2/src/chunk_dispatch_state.c:85
#4 0x0000000000628b1b in ExecProcNode (node=0x13df280) at ../../../src/include/executor/executor.h:247
#5 ExecModifyTable (pstate=0x13ded28) at nodeModifyTable.c:2025
#6 0x000000000060682a in ExecProcNode (node=0x13de948) at ../../../src/include/executor/executor.h:247
#7 ExecutePlan (execute_once=, dest=0xcd7020 , direction=, numberTuples=0, sendTuples=false, operation=CMD_INSERT,
use_parallel_mode=, planstate=0x13de948, estate=0x13de480) at execMain.c:1723
#8 standard_ExecutorRun (queryDesc=0x147e210, direction=, count=0, execute_once=) at execMain.c:364
#9 0x00007fc1e8e4a2f5 in pgss_ExecutorRun (queryDesc=0x147e210, direction=ForwardScanDirection, count=0, execute_once=) at pg_stat_statements.c:892
#10 0x000000000073e90a in ProcessQuery (plan=,
sourceText=0x143e608 "insert into t_mpp_11072_1d_yvodkg_0(entity_id,time,ind_0b7s1nvgx,ind_041rf6nak,ind_0v3de6ot,ind_0c1l03190,ind_0a2wvh950,ind_0dren41g,ind_0cs2gmsd1,ind_0bm4i2zmd,ind_06orqb92t,ind_0awnlmiwk,ind_0djyvh5"..., params=0x0, queryEnv=0x0, dest=0xcd7020 , completionTag=0x7ffe9dae5540 "") at pquery.c:161
#11 0x000000000073eb31 in PortalRunMulti (portal=portal@entry=0x13a1510, isTopLevel=isTopLevel@entry=true, setHoldSnapshot=setHoldSnapshot@entry=false,
dest=0xcd7020 , dest@entry=0x12cc160, altdest=0xcd7020 , altdest@entry=0x12cc160, completionTag=completionTag@entry=0x7ffe9dae5540 "")
at pquery.c:1286
#12 0x000000000073f5ec in PortalRun (portal=portal@entry=0x13a1510, count=count@entry=9223372036854775807, isTopLevel=isTopLevel@entry=true, run_once=,
dest=dest@entry=0x12cc160, altdest=altdest@entry=0x12cc160, completionTag=completionTag@entry=0x7ffe9dae5540 "") at pquery.c:799
#13 0x000000000073d0fd in exec_execute_message (max_rows=9223372036854775807, portal_name=0x12cbd50 "") at postgres.c:2036
#14 PostgresMain (argc=, argv=argv@entry=0x13250d8, dbname=0x1325020 "ies_db", username=) at postgres.c:4245
#15 0x000000000048e61f in BackendRun (port=0x131a140) at postmaster.c:4361
#16 BackendStartup (port=0x131a140) at postmaster.c:4033
#17 ServerLoop () at postmaster.c:1706
#18 0x00000000006d3bfa in PostmasterMain (argc=argc@entry=1, argv=argv@entry=0x12c5d80) at postmaster.c:1379
#19 0x000000000048f06f in main (argc=1, argv=0x12c5d80) at main.c:228
(gdb) l
465 ListCell *lc;
466
467 state->arbiter_indexes = NIL;
468 foreach (lc, dispatch->arbiter_indexes)
469 {
470 Oid hypertable_index = lfirst_oid(lc);
471 Chunk *chunk = ts_chunk_get_by_relid(RelationGetRelid(chunk_rel), 0, true);
472 ChunkIndexMapping cim;
473
474 if (ts_chunk_index_get_by_hypertable_indexrelid(chunk, hypertable_index, &cim) < 1)

[Specter-Y]

This time, after the ts_chunk_insert_state_create is called. but chunk_insert_state_set_arbiter_indexes is crash

[Specter-Y]

dear cevian, please help us find the cause of the problem. tks a lot.

[JLockerman]

@Specter-Y Not quite enough info here to debug, could you run fr v in your gdb instance?
Additionally, what are the actual commands you're running to get the segfault?

[cevian]

@Specter-Y We were able to reproduce this bug. We are working on a fix now. I'll update you with our progress. Thank you.