Bigquery Row-Level Deletes + Erase After on Database Collections #5293

pattisdr · 2024-09-17T23:02:33Z

Closes #PROD-2744
Closes #PROD-2745

Description Of Changes

Adds initial support for collection-level masking overrides via Bigquery and
extends erase_after to work for database connectors not just saas connectors.

...
   collections:
      - name: address
        fides_meta:
          erase_after: [other_dataset.other_collection]
          masking_strategy_override:
            strategy: delete

Code Changes

Specifying a delete masking strategy override at the collection level on a bigquery dataset yaml will cause matching rows to be deleted instead of updated when running an erasure request. The collection-level override takes precedence over the policy override. Again this just takes affect for Bigquery only as a first iteration.
Extends erase_after functionality to be incorporated into the graph if specified for a dataset connector, not just a saas connector. This lets customers specify they want a specific node to run after another if necessary, while there is no intelligent ordering for erasure deletes.

Caution

Collections need FK's to facilitate deletes (and updates for that matter)
Erasure order is unchanged. If you require that one node be erased before another, use erase_after functionality
masking_strategy_overrrides at the collection level are ignored for all but bigquery integrations

Steps to Confirm

Testing instructions here: https://ethyca.atlassian.net/wiki/spaces/EN/pages/3292397569/Bigquery+Row-Level+Deletes

Pre-Merge Checklist

vercel · 2024-09-17T23:02:37Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
fides-plus-nightly	⬜️ Ignored (Inspect)	Visit Preview		Sep 25, 2024 0:37am

cypress · 2024-09-17T23:16:02Z

fides Run #10122

Run Properties: Passed #10122 • d8b0787ae5 ℹ️: Merge 992f9e9e989dc9d7a892b485c818a4b7c073b5c0 into 155e1fd8bcf545599af273b06a62...

Project	`fides`
Branch Review	`refs/pull/5293/merge`
Run status	`Passed #10122`
Run duration	`00m 39s`
Commit	`d8b0787ae5 ℹ️: Merge 992f9e9e989dc9d7a892b485c818a4b7c073b5c0 into 155e1fd8bcf545599af273b06a62...`
Committer	`Dawn Pattison`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

… bigquery - Seed existing bigquery table with employee record - Updated employee dataset reference direction - Confirm employee record exists before running DSR and confirm it's been removed afterwards

…ment built.

…saas connectors.

…just use the one.

data/dataset/bigquery_delete_override_test_dataset.yml

data/dataset/bigquery_example_test_dataset.yml

requirements.txt

src/fides/api/service/connectors/query_config.py

pattisdr · 2024-09-23T11:11:07Z

src/fides/api/service/connectors/query_config.py

+    def generate_delete(self, row: Row, client: Engine) -> Optional[Delete]:
+        """Returns a SQLAlchemy DELETE statement for BigQuery. Does not actually execute the delete statement.
+
+        Used when a collection-level masking override is present and the masking strategy is DELETE.
+        """
+        non_empty_primary_keys: Dict[str, Field] = filter_nonempty_values(
+            {
+                fpath.string_path: fld.cast(row[fpath.string_path])
+                for fpath, fld in self.primary_key_field_paths.items()
+                if fpath.string_path in row
+            }
+        )
+
+        valid = len(non_empty_primary_keys) > 0
+        if not valid:
+            logger.warning(
+                "There is not enough data to generate a valid DELETE statement for {}",
+                self.node.address,
+            )
+            return None
+
+        table = Table(
+            self.node.address.collection, MetaData(bind=client), autoload=True
+        )
+        pk_clauses: List[ColumnElement] = [
+            getattr(table.c, k) == v for k, v in non_empty_primary_keys.items()
+        ]
+        return table.delete().where(*pk_clauses)
+


New method for Bigquery only to start to generate a DELETE statement for the relevant primary keys.

src/fides/api/service/connectors/sql_connector.py

tests/ops/integration_tests/setup_scripts/postgres_setup.py

tests/ops/service/privacy_request/test_request_runner_service.py

pattisdr · 2024-09-23T21:18:08Z

Lots of failing tests, on first glance a lot aren't related, but let me go through and double check them -

increased failures in saas connector tests - I wonder if erase_after could be interacting? 👀 looking

EDIT: SaaS Connectors were flaky tests

erosselli

Nice work! Tested manually and it looks good

tests/ops/service/connectors/test_queryconfig.py

# Conflicts: # CHANGELOG.md

pattisdr · 2024-09-25T02:05:02Z

Merging, I believe the failing tests are also on main

cypress · 2024-09-25T02:20:45Z

fides Run #10128

Run Properties: Passed #10128 • 0e2db7c28d: Bigquery Row-Level Deletes + Erase After on Database Collections (#5293)

Project	`fides`
Branch Review	`main`
Run status	`Passed #10128`
Run duration	`00m 38s`
Commit	`0e2db7c28d: Bigquery Row-Level Deletes + Erase After on Database Collections (#5293)`
Committer	`Dawn Pattison`
View all properties for this run ↗︎

Test results
Failures	`0`
Flaky	`0`
Pending	`0`
Skipped	`0`
Passing	`4`
⚠️ You've recorded test results over your free plan limit. Upgrade your plan to view test results.
View all changes introduced in this branch ↗︎

pattisdr added 6 commits September 10, 2024 18:08

POC - executing a delete request on a row in a postgres db.

65dfd23

Merge main

d8d1cb1

Merge branch 'main' into PROD-2744-row-deletes

4aeefe5

Generate a delete statement for bigquery -

e07c481

Revert sql-connector code for now to focus on bigquery-

e358379

Remove masking strategy override on postgres dataset

e7f22d7

pattisdr mentioned this pull request Sep 17, 2024

POC Postgres Row-Level Deletes #5276

Closed

13 tasks

pattisdr added 3 commits September 19, 2024 15:03

Add end-to-end testing of row-level deletes as a masking strategy for…

9fa25f2

… bigquery - Seed existing bigquery table with employee record - Updated employee dataset reference direction - Confirm employee record exists before running DSR and confirm it's been removed afterwards

Add query config tests for bigquery around the update or delete state…

0f60402

…ment built.

Improve docstrings

4973996

pattisdr marked this pull request as ready for review September 19, 2024 22:08

pattisdr added the run unsafe ci checks Runs fides-related CI checks that require sensitive credentials label Sep 19, 2024

pattisdr added 2 commits September 21, 2024 03:12

Extend "erase_after" concept so it can be used by databases not just …

c67acc4

…saas connectors.

Mark tests using bigquery

2bc6a90

pattisdr changed the title ~~Bigquery Row-Level Deletes~~ Bigquery Row-Level Deletes + Erase After on Database Collections Sep 23, 2024

Merge branch 'main' into PROD-2744-row-deletes

921f169

vercel bot deployed to Preview September 23, 2024 09:20 View deployment

pattisdr added 3 commits September 23, 2024 05:56

Refactor tests to get rid of the second bigquery example dataset, to …

97a600b

…just use the one.

Remove unnecessary statement.

aea2445

Update CHANGELOG and remove breakpoints

7066780

pattisdr commented Sep 23, 2024

View reviewed changes

Make tests repeatable so the order doesn't matter

f731c73

erosselli self-requested a review September 23, 2024 21:13

erosselli approved these changes Sep 23, 2024

View reviewed changes

tests/ops/service/connectors/test_queryconfig.py Outdated Show resolved Hide resolved

tests/ops/service/connectors/test_queryconfig.py Outdated Show resolved Hide resolved

pattisdr added 3 commits September 24, 2024 16:07

Merge branch 'main' into PROD-2744-row-deletes

b3af3f0

# Conflicts: # CHANGELOG.md

Bump fideslang commit.

1879694

Merge branch 'main' into PROD-2744-row-deletes

825d104

# Conflicts: # CHANGELOG.md

vercel bot deployed to Preview September 24, 2024 21:14 View deployment

pattisdr added 2 commits September 24, 2024 18:15

Fix copy/paste issues

e425190

Merge branch 'main' into PROD-2744-row-deletes

992f9e9

# Conflicts: # CHANGELOG.md

pattisdr merged commit 0e2db7c into main Sep 25, 2024
39 of 45 checks passed

pattisdr deleted the PROD-2744-row-deletes branch September 25, 2024 02:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bigquery Row-Level Deletes + Erase After on Database Collections #5293

Bigquery Row-Level Deletes + Erase After on Database Collections #5293

pattisdr commented Sep 17, 2024 •

edited

Loading

vercel bot commented Sep 17, 2024 •

edited

Loading

cypress bot commented Sep 17, 2024 •

edited

Loading

pattisdr Sep 23, 2024

pattisdr commented Sep 23, 2024 •

edited

Loading

erosselli left a comment

pattisdr commented Sep 25, 2024

cypress bot commented Sep 25, 2024

Bigquery Row-Level Deletes + Erase After on Database Collections #5293

Bigquery Row-Level Deletes + Erase After on Database Collections #5293

Conversation

pattisdr commented Sep 17, 2024 • edited Loading

Description Of Changes

Code Changes

Caution

Steps to Confirm

Pre-Merge Checklist

vercel bot commented Sep 17, 2024 • edited Loading

cypress bot commented Sep 17, 2024 • edited Loading

fides Run #10122

pattisdr Sep 23, 2024

Choose a reason for hiding this comment

pattisdr commented Sep 23, 2024 • edited Loading

erosselli left a comment

Choose a reason for hiding this comment

pattisdr commented Sep 25, 2024

cypress bot commented Sep 25, 2024

fides Run #10128

pattisdr commented Sep 17, 2024 •

edited

Loading

vercel bot commented Sep 17, 2024 •

edited

Loading

cypress bot commented Sep 17, 2024 •

edited

Loading

pattisdr commented Sep 23, 2024 •

edited

Loading