Fix FK constraints for compressed chunks

[svenklemm]

When foreign key support for compressed chunks was added we moved
the FK constraint from the uncompressed chunk to the compressed chunk as
part of compress_chunk and moved it back as part of decompress_chunk.
With the addition of partially compressed chunks in 2.10.x this approach
was no longer sufficient and the FK constraint needs to be present on
both the uncompressed and the compressed chunk.

While this patch will fix future compressed chunks a migration has to be
run after upgrading timescaledb to migrate existing chunks affected by
this.

The following code will fix any affected hypertables:

CREATE OR REPLACE FUNCTION pg_temp.constraint_columns(regclass, int2[]) RETURNS text[] AS
$$
  SELECT array_agg(attname) FROM unnest($2) un(attnum) LEFT JOIN pg_attribute att ON att.attrelid=$1 AND att.attnum = un.attnum;
$$ LANGUAGE SQL SET search_path TO pg_catalog, pg_temp;

DO $$
DECLARE
  ht_id int;
  ht regclass;
  chunk regclass;
  con_oid oid;
  con_frelid regclass;
  con_name text;
  con_columns text[];
  chunk_id int;

BEGIN

  -- iterate over all hypertables that have foreign key constraints
  FOR ht_id, ht in
    SELECT
      ht.id,
      format('%I.%I',ht.schema_name,ht.table_name)::regclass
    FROM _timescaledb_catalog.hypertable ht
    WHERE
      EXISTS (
        SELECT FROM pg_constraint con
        WHERE
          con.contype='f' AND
          con.conrelid=format('%I.%I',ht.schema_name,ht.table_name)::regclass
      )
  LOOP
    RAISE NOTICE 'Hypertable % has foreign key constraint', ht;

    -- iterate over all foreign key constraints on the hypertable
    -- and check that they are present on every chunk
    FOR con_oid, con_frelid, con_name, con_columns IN
      SELECT con.oid, con.confrelid, con.conname, pg_temp.constraint_columns(con.conrelid,con.conkey)
      FROM pg_constraint con
      WHERE
        con.contype='f' AND
        con.conrelid=ht
    LOOP
      RAISE NOTICE 'Checking constraint % %', con_name, con_columns;
      -- check that the foreign key constraint is present on the chunk

      FOR chunk_id, chunk IN
        SELECT
          ch.id,
          format('%I.%I',ch.schema_name,ch.table_name)::regclass
        FROM _timescaledb_catalog.chunk ch
        WHERE
          ch.hypertable_id=ht_id
      LOOP
        RAISE NOTICE 'Checking chunk %', chunk;
        IF NOT EXISTS (
          SELECT FROM pg_constraint con
          WHERE
            con.contype='f' AND
            con.conrelid=chunk AND
            con.confrelid=con_frelid  AND
            pg_temp.constraint_columns(con.conrelid,con.conkey) = con_columns
        ) THEN
          RAISE WARNING 'Restoring constraint % on chunk %', con_name, chunk;
          PERFORM _timescaledb_functions.constraint_clone(con_oid, chunk);
          INSERT INTO _timescaledb_catalog.chunk_constraint(chunk_id, dimension_slice_id, constraint_name, hypertable_constraint_name) VALUES (chunk_id, NULL, con_name, con_name);
        END IF;

      END LOOP;
    END LOOP;

  END LOOP;

END
$$;

DROP FUNCTION pg_temp.constraint_columns(regclass, int2[]);

Fixes #6759

[antekresic]

LGTM

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 80.93%. Comparing base (59f50f2) to head (5586d8f).
Report is 110 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #6797      +/-   ##
==========================================
+ Coverage   80.06%   80.93%   +0.86%     
==========================================
  Files         190      197       +7     
  Lines       37181    37015     -166     
  Branches     9450     9661     +211     
==========================================
+ Hits        29770    29957     +187     
- Misses       2997     3202     +205     
+ Partials     4414     3856     -558     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[timescale-automation]

Automated backport to 2.14.x not done: cherry-pick failed.

Git status

HEAD detached at origin/2.14.x
You are currently cherry-picking commit ecf6beae5.
  (fix conflicts and run "git cherry-pick --continue")
  (use "git cherry-pick --skip" to skip this patch)
  (use "git cherry-pick --abort" to cancel the cherry-pick operation)

Changes to be committed:
	new file:   .unreleased/pr_6797
	modified:   src/chunk.h
	modified:   tsl/src/compression/api.c
	modified:   tsl/test/expected/compression.out
	new file:   tsl/test/expected/compression_fks.out
	modified:   tsl/test/sql/CMakeLists.txt
	new file:   tsl/test/sql/compression_fks.sql

Unmerged paths:
  (use "git add <file>..." to mark resolution)
	both modified:   src/chunk.c

Job log

[SystemParadox]

Why isn't this being done automatically as part of the upgrade? Manual migration like this is a real pain.

[eyadmba]

Does this script need to be run if I don't have any compressed hypertables? If not, then the docs should mention that. Right now the docs say that the script "must" be run after the ALTER EXTENSION.

[mac866]

Hi,

the script creates and immediately deletes the function constraint_columns.
Where, when and how (parameters?) is constraint_columns called?

Thank you very much!

[adschmu]

@svenklemm Release docs say "for this release only". Does this mean that
A) we will get an upgrade script for this at some point (e.g. with 2.16.x),
or does it mean that
B) we will continue to have to run this whenever we upgrade coming from < 2.15.0 to >= 2.15.0, so we never get it integrated?

[svenklemm]

Why isn't this being done automatically as part of the upgrade? Manual migration like this is a real pain.

There are pros and cons to both whenever reasonable we try to have these scripts run as part of normal upgrade. In this case we decided not to do it because of worst case runtime of the script. In the worst case this script can run for a long time and we want extension updates to be fast thats why we decided to have it separate. In the normal case the script should finish within seconds and is also idempotent.

Does this script need to be run if I don't have any compressed hypertables? If not, then the docs should mention that. Right now the docs say that the script "must" be run after the ALTER EXTENSION.

No it doesnt, it also doesnt need to be run when you dont have foreign keys. But listing all those constraints is tedious and since in those cases the script just does nothing it's easier to always run it.

the script creates and immediately deletes the function constraint_columns.
Where, when and how (parameters?) is constraint_columns called?

constraint_columns is called on line 39 and 63 of the script in the DO block.

Release docs say "for this release only". Does this mean that
A) we will get an upgrade script for this at some point (e.g. with 2.16.x),
or does it mean that
B) we will continue to have to run this whenever we upgrade coming from < 2.15.0 to >= 2.15.0, so we never get it integrated?

It is the latter, you have to run this whenever you are upgrading from < 2.15.0.