Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Incrementally Updating Partitions of a Table Without Locking Parent Table on Postgres #577

Open
magidandrew opened this issue Nov 1, 2024 · 1 comment

Comments

@magidandrew
Copy link

I’m working with postgres on a table that uses partitioning to manage multi-tenancy for our clients. Each client’s data is stored in a partitioned table, allowing our Hasura Graphql API to expose data for all clients through a single parent table.

Here's a snippet of the dbt model:

{{ config(
    materialized='incremental',
    pre_hook="TRUNCATE TABLE dagster.computed_achievements_{{ var('client_season') }};",
    alias='computed_achievements',
    tags=['1.0.0']
) }}

WITH selected_client_season AS (
    -- Query logic here
)

Our goal is to refresh a specific partition (based on client_season) without locking the parent table, so other clients’ data remains accessible during the operation. Currently, using TRUNCATE TABLE on a partition seems to lock the parent table, which can disrupt access for other clients.

So my question is:
Is there a way to truncate or refresh a partitioned table in postgres without locking the parent table? We want to perform this operation efficiently without impacting other partitions or client access.

It feels like the method above is an anti-pattern so we're open to suggestions!

@matsonj
Copy link

matsonj commented Nov 1, 2024

Delete & Insert with the partition selected may work. This is a SQL server project though.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants