Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Task/rdmp 32 regex redaction #2009

Merged
merged 87 commits into from
Nov 27, 2024
Merged

Task/rdmp 32 regex redaction #2009

merged 87 commits into from
Nov 27, 2024

Conversation

JFriel
Copy link
Collaborator

@JFriel JFriel commented Sep 26, 2024 •
edited
Loading

Proposed Change

Adds the ability to perform regex redactions on incoming data loads and catalogues.
Only supports MSSQL currently

Type of change

What types of changes does your code introduce? Tick all that apply.

  • Bugfix (non-breaking change which fixes an issue)
  • New Feature (non-breaking change which adds functionality)
  • Breaking Change (fix or feature that would cause existing functionality to not work as expected)
  • Documentation-Only Update
  • Other (if none of the other choices apply)

Checklist

By opening this PR, I confirm that I have:

  • Ensured that the PR branch is in sync with the target branch (i.e. it is automatically merge-able)
  • Created or updated any tests if relevant
  • Have validated this change against the Test Plan
  • Requested a review by one of the repository maintainers
  • Have written new documentation or updated existing documentation to detail any new or updated functionality and how to use it
  • Have added an entry into the changelog

@JFriel JFriel marked this pull request as ready for review October 28, 2024 13:59
@JFriel JFriel requested review from a team and jas88 October 28, 2024 14:00
JFriel and others added 13 commits October 29, 2024 10:49
@JFriel
fix typos
34115c2
@JFriel
Merge branch 'release/8.4.0' of https://github.com/HicServices/RDMP i…
7d84d59 
…nto task/RDMP-32-regex-redaction
@JFriel
tidy up tests
81077e5
@JFriel
tidy up tests
8f5a2fd
@JFriel
Merge branch 'release/8.4.0' into task/RDMP-32-regex-redaction
81faa2f
@jas88
Include regex redaction doc in VS Solution file
fecd525
@jas88
Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs
231ad98 
Syntax cleanup
@jas88
Merge branch 'develop' into task/RDMP-32-regex-redaction
3f7fec6
@jas88
Update ThrowImmediatelyActivator.cs
61ba1d0 
Remove obsolete comment
@jas88
Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs
9a969d2 
Start being green and recycling
@jas88
Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs
07df2ad 
Finish recycling test objects
@JFriel
Merge branch 'release/8.4.0' into task/RDMP-32-regex-redaction
641ddfc
@JFriel
Merge branch 'release/8.4.0' into task/RDMP-32-regex-redaction
13accf1
@JFriel JFriel merged commit 090097e into release/8.4.0 Nov 27, 2024
5 checks passed
@JFriel JFriel deleted the task/RDMP-32-regex-redaction branch November 27, 2024 12:42
JFriel added a commit that referenced this pull request Nov 29, 2024
@JFriel @rdteviotdale
@jas88 @bpeacock001 @JBaird00183 @dependabot
Release: 8.4.0 (#2006)
b1457d0 
* add changelog

* Bugfix/rdmp 253 filter ordering (#2007)

* add start

* partial

* tidy up code

* filter updates

* update db migrations

* update correct db

* update changelog

* add base creation sql

* fix create

* tidy up

* update insert

* tidy up

* interim

* style updates

* add some info

* start of where clause

* improved

* improved dates

* improve bookends

* add changelog

* update changelog

* improve ui with frequency

* tidy up for tests

* add summary

* rename chart

* interim people

* add people label

* better where

* move items around

* fix up tests

* tidy up code

* tidy up

* made db independant

* add changelog

* add missing file

* Bugfix/rdmp 259 delta load off by one issue (#2024)

* fix off by one issue

* update tests

* update changelog

* check for multi-server query

* update changelog

* update changelog

* Add override for RAW table date column in delta loads (#2052)

* fix table attacher special column issue

* add override

* add extra override

* fix tests

* tidy up

* add docs

* update deps

* add safe ref

* interim

* working check

* tidy up

* add changelog

* fix typos

* update typo

* fix typos

* fix typos

* Changes made to popup buttons (#2076)

* Changes made to popup buttons

* Bump the aws-sdk group with 4 updates (#2074)

Bumps the aws-sdk group with 4 updates: [AWSSDK.S3](https://github.com/aws/aws-sdk-net), [AWSSDK.SecurityToken](https://github.com/aws/aws-sdk-net), [AWSSDK.SSO](https://github.com/aws/aws-sdk-net) and [AWSSDK.SSOOIDC](https://github.com/aws/aws-sdk-net).


Updates `AWSSDK.S3` from 3.7.407 to 3.7.407.1
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SecurityToken` from 3.7.401.5 to 3.7.401.6
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SSO` from 3.7.400.56 to 3.7.400.57
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

Updates `AWSSDK.SSOOIDC` from 3.7.400.56 to 3.7.400.57
- [Release notes](https://github.com/aws/aws-sdk-net/releases)
- [Changelog](https://github.com/aws/aws-sdk-net/blob/main/SDK.CHANGELOG.MD)
- [Commits](https://github.com/aws/aws-sdk-net/commits)

---
updated-dependencies:
- dependency-name: AWSSDK.S3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SecurityToken
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SSO
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
- dependency-name: AWSSDK.SSOOIDC
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: aws-sdk
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Update CHANGELOG.md

* Tidied up code

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Task/rdmp 32 regex redaction (#2009)

* interim

* interim

* redact and add some tests

* add missing interface

* add regex redaction keys

* add helper

* add concept

* update interfaces

* add icons and fix restore

* add missing files

* add docs

* add regions

* add ignore

* add speed test

* improve speed

* improve speed

* update padding

* stress test

* add test case

* add fk names

* faster data load

* faster update

* more timers

* super fast redaction

* tidy up code

* woking keys

* improved

* add limiting

* shared helper

* shared

* interim

* working

* add summary

* restore tests working

* add tests

* start og ui update

* start of ui

* basic configuraion

* working add

* user interface

* update for tests

* update test

* tests

* tidy up

* updates

* fix merge

* ad dicons

* add missing icons

* tidyup

* fix tests

* add missing file

* update migration number

* add changelog

* add timeout

* fix build

* interim

* tidy up

* tidy up code

* add publish

* add todo

* add redact all

* add column

* update table

* add docs

* update markdown

* fix typo

* fix typos

* tidy up tests

* tidy up tests

* Include regex redaction doc in VS Solution file

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Syntax cleanup

* Update ThrowImmediatelyActivator.cs

Remove obsolete comment

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Start being green and recycling

* Update ExecuteCommandPerformRegexRedactionOnCatalogueTests.cs

Finish recycling test objects

---------

Co-authored-by: James A Sutherland <j@sutherland.pw>

* codeql updates

* update client xml

* code ql updates

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: rdteviotdale <r.d.teviotdale@dundee.ac.uk>
Co-authored-by: James A Sutherland <j@sutherland.pw>
Co-authored-by: Brian <76164974+bpeacock001@users.noreply.github.com>
Co-authored-by: JBaird00183 <109660435+JBaird00183@users.noreply.github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rdteviotdale rdteviotdale rdteviotdale approved these changes

@jas88 jas88 Awaiting requested review from jas88

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@JFriel @rdteviotdale @jas88