Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: JSON Masking #1899

Open
wants to merge 11 commits into
base: dev
Choose a base branch
from
Open

feat: JSON Masking #1899

wants to merge 11 commits into from

Conversation

jamfor352
Copy link

@jamfor352 jamfor352 commented Aug 10, 2024

Hi!

AKHQ is fantastic, but the data masking using regex doesn't really scale well if you have complex personal data (pii) requirements (for example, if you need to mask all fields by default apart from carefully selected fields in a production environment). Crafting a regex for this can be flaky. Especially if you need to deal with multiple lines and account for integers, arrays, all sorts of data types. Or if you want to be able to expose some fields on some topics, but not others, etc.

I've been trying my fork locally which introduces a new JSON filtering system (while keeping the regex stuff entirely in place) and it works great, and I'd like to contribute it upstream. I think I've added more than sufficient tests as well, and retained backwards compatibility by defaulting to using the regex filtering if not defined (which current users won't, since mode is a new property).

It has two modes - json_mask_by_default (where everything is masked by default and you have to specifically opt in to see specific fields - useful for regulation-heavy environments) and json_show_by_default where everything shows by default but you can mask specific bits of data.

@jamfor352
Copy link
Author

jamfor352 commented Aug 11, 2024

I just pushed a commit fixing datetime parsing in AvroSerializer as I realised it led to some flaky tests as well. Not related to my changes here, but a nice to have (just noticed it when checking the workflow results)

@AlexisSouquiere
Copy link
Collaborator

@jamfor352 I will take time to review it and give you my feedback next week

@emerfan
Copy link

emerfan commented Sep 18, 2024

Hi, have been using this build locally and it's awesome - following the PR

@jamfor352
Copy link
Author

Hi, have been using this build locally and it's awesome - following the PR

Thank you :) Glad to hear it's worked well for you!

@AlexisSouquiere
Copy link
Collaborator


public Record maskRecord(Record record) {
try {
if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should test for record.getValue() != null first (for tombstones) to avoid generating an exception (even if this exception is catch just after)


public Record maskRecord(Record record) {
try {
if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should test for record.getValue() != null first (for tombstones) to avoid generating an exception (even if this exception is catch just after)

Comment on lines 23 to 46
JsonMaskingFilter foundFilter = null;
for (JsonMaskingFilter filter : jsonMaskingFilters) {
if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
foundFilter = filter;
}
}
if (foundFilter != null) {
return applyMasking(record, foundFilter.getKeys());
} else {
return applyMasking(record, List.of());
}
} else {
return record;
}
} catch (Exception e) {
LOG.error("Error masking record", e);
return record;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To improve readability, what do you think of this ?

        if (record.getValue() == null) {
            return record;
        }

        try {
            jsonMaskingFilters.stream()
                .filter(filter -> record.getTopic().getName().equalsIgnoreCase(filter.getTopic()))
                .findFirst()
                .ifPresentOrElse(filter -> applyMasking(record, filter.getKeys()).getValue(),
                    () -> applyMasking(record, List.of()).getValue());
        } catch (Exception e) {
            LOG.error("Error masking record", e);
        }

        return record;

Comment on lines 22 to 39
JsonMaskingFilter foundFilter = null;
for (JsonMaskingFilter filter : jsonMaskingFilters) {
if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
foundFilter = filter;
}
}
if (foundFilter != null) {
return applyMasking(record, foundFilter.getKeys());
} else {
return record;
}
} else {
return record;
}
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can improve it like my previous comment (just using ifPresent and the first parameter)

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

MaskerFactory can be removed by adding @requires annotation on the different maskers

Comment on lines 14 to 18
@RequiredArgsConstructor
public class JsonMaskByDefaultMasker implements Masker {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using this to create the masker only if we have the json_mask_by_default property will follow what we already have at some other places. And we won't need the Factory anymore :)

@Singleton
@Requires(property = "akhq.security.data-masking.mode", value = "json_mask_by_default")
@RequiredArgsConstructor
public class JsonMaskByDefaultMasker implements Masker {

    @Inject
    private final DataMasking dataMasking;

And then use dataMasking.getJsonFilters(), etc.


import org.akhq.models.Record;

public class NoOpMasker implements Masker {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adding a default value will instantiate this Masker if there is no value or 'none'

@Singleton
@Requires(property = "akhq.security.data-masking.mode", value = "none", defaultValue = "none")

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@AlexisSouquiere This is the only change that was requested that I adjusted a little - I actually defaulted it to the Regex one for backwards compatibility for current users (ie they won't upgrade, and suddenly their filters stop working). The regex filter currently runs all the time by default anyway. Didn't want this to be a breaking change

import static org.junit.jupiter.api.Assertions.assertInstanceOf;

@MicronautTest(environments = "json-mask-by-default-data-masking")
class JsonMaskByDefaultMaskerTest extends MaskerTestHelper {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add a test for record with a null value

@AlexisSouquiere
Copy link
Collaborator

AlexisSouquiere commented Sep 26, 2024

@jamfor352 thanks for the PR it's really interesting 😄
Can you check the different comments I put ? Don't hesitate if you have a question !

@jamfor352
Copy link
Author

@jamfor352 thanks for the PR it's really interesting 😄
Can you check the different comments I put ? Don't hesitate if you have a question !

Thanks, they all seem reasonable - will get around to it sometime this week.

@jamfor352
Copy link
Author

Will update in the coming days - been busy moving house so haven't had much free time!

@emerfan
Copy link

emerfan commented Oct 24, 2024

Keeping it fresh in minds - I'm very interested in deploying this build, I can obviously pull it down and make the changes myself but would prefer to use the master build

@jamfor352
Copy link
Author

Keeping it fresh in minds - I'm very interested in deploying this build, I can obviously pull it down and make the changes myself but would prefer to use the master build

Hey, definitely. Was planning to do this last week but quite a lot came up, so wasn't able to get around to it. I've set some time aside specifically for this PR this evening (7pm UK time) so should be all good.

@jamfor352
Copy link
Author

Hi @AlexisSouquiere I've updated my PR based on recommendations (with one small difference, re: default masking filter, to not break current compatibility). Added documentation, too.

I've also made some other improvements that I've spotted - made the filtering more consistent (ie, always using qualified paths for both JSON maskers), and also some slight performance improvements.

cc: @emerfan could you try this new build out and confirm everything is working okay for you as well after the changes? Please note the change to json_mask_by_default where nested objects are now distinct from each other (eg metadata.value and address.value need separate entries, rather than just value) to ensure it still works 👍

@emerfan
Copy link

emerfan commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

@jamfor352
Copy link
Author

jamfor352 commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

@emerfan
Copy link

emerfan commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

Here is a close sample, there's nothing complex about it though... I've tested this value in the unit tests and it also passes.

            {
              "Product": "Cheese",
              "Id": 12345,
              "SellByDate": "2024-01-11T00:00:00+00:00",
              "IsDelicious": true
            }

Have you tried running it and testing with the UI / Kafka?

If I replace the updated maskRecord with your original one, it works fine. I can debug to the point I can see the masked record returned to RecordRepository but not as easy to see from there what's happening

    public Record maskRecord(Record record) {
        try {
            if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
                JsonMaskingFilter foundFilter = null;
                for (JsonMaskingFilter filter : jsonMaskingFilters) {
                    if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
                        foundFilter = filter;
                    }
                }
                if (foundFilter != null) {
                    return applyMasking(record, foundFilter.getKeys());
                } else {
                    return applyMasking(record, List.of());
                }
            } else {
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record", e);
            return record;
        }
    }

@jamfor352
Copy link
Author

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

Here is a close sample, there's nothing complex about it though... I've tested this value in the unit tests and it also passes.

            {
              "Product": "Cheese",
              "Id": 12345,
              "SellByDate": "2024-01-11T00:00:00+00:00",
              "IsDelicious": true
            }

Have you tried running it and testing with the UI / Kafka?

If I replace the updated maskRecord with your original one, it works fine. I can debug to the point I can see the masked record returned to RecordRepository but not as easy to see from there what's happening

    public Record maskRecord(Record record) {
        try {
            if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
                JsonMaskingFilter foundFilter = null;
                for (JsonMaskingFilter filter : jsonMaskingFilters) {
                    if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
                        foundFilter = filter;
                    }
                }
                if (foundFilter != null) {
                    return applyMasking(record, foundFilter.getKeys());
                } else {
                    return applyMasking(record, List.of());
                }
            } else {
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record", e);
            return record;
        }
    }

Can you provide your full application.yml, if possible?

@emerfan
Copy link

emerfan commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

Here is a close sample, there's nothing complex about it though... I've tested this value in the unit tests and it also passes.

            {
              "Product": "Cheese",
              "Id": 12345,
              "SellByDate": "2024-01-11T00:00:00+00:00",
              "IsDelicious": true
            }

Have you tried running it and testing with the UI / Kafka?
If I replace the updated maskRecord with your original one, it works fine. I can debug to the point I can see the masked record returned to RecordRepository but not as easy to see from there what's happening

    public Record maskRecord(Record record) {
        try {
            if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
                JsonMaskingFilter foundFilter = null;
                for (JsonMaskingFilter filter : jsonMaskingFilters) {
                    if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
                        foundFilter = filter;
                    }
                }
                if (foundFilter != null) {
                    return applyMasking(record, foundFilter.getKeys());
                } else {
                    return applyMasking(record, List.of());
                }
            } else {
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record", e);
            return record;
        }
    }

Can you provide your full application.yml, if possible?

akhq:
  connections:
    local:
      properties:
        bootstrap.servers: "localhost:9092"
  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Product

@jamfor352
Copy link
Author

jamfor352 commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

Here is a close sample, there's nothing complex about it though... I've tested this value in the unit tests and it also passes.

            {
              "Product": "Cheese",
              "Id": 12345,
              "SellByDate": "2024-01-11T00:00:00+00:00",
              "IsDelicious": true
            }

Have you tried running it and testing with the UI / Kafka?
If I replace the updated maskRecord with your original one, it works fine. I can debug to the point I can see the masked record returned to RecordRepository but not as easy to see from there what's happening

    public Record maskRecord(Record record) {
        try {
            if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
                JsonMaskingFilter foundFilter = null;
                for (JsonMaskingFilter filter : jsonMaskingFilters) {
                    if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
                        foundFilter = filter;
                    }
                }
                if (foundFilter != null) {
                    return applyMasking(record, foundFilter.getKeys());
                } else {
                    return applyMasking(record, List.of());
                }
            } else {
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record", e);
            return record;
        }
    }

Can you provide your full application.yml, if possible?

akhq:
  connections:
    local:
      properties:
        bootstrap.servers: "localhost:9092"
  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Product

I'm wondering if it's something to do with the "disable if no filters found" (which I have realised probably shouldnt be in the mask by default mode) nd there being a bug with determining this, or perhaps the JSON parsing

Thanks for the YAML - I'll check it out later.

@emerfan
Copy link

emerfan commented Nov 3, 2024

Hi @jamfor352 this isn't working for me .... debugging I can see that masking is being applied for the record (I'm using json_mask_by_default, but the unmasked record is being returned to the UI. Sanity checked by dropping your original try/catch logic in to the maskRecord method and that's still working. I'm guessing something funky up with the RecordRepository / how it's returning the record maybe - I'll get back to it later and dig further

  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Results

Could you provide an example record for me to try out with my end? All the tests are passing, which is odd based on how the masker is activated. In theory the logic hasnt changed either beyond the nested field adjustment. Will take another look.

Here is a close sample, there's nothing complex about it though... I've tested this value in the unit tests and it also passes.

            {
              "Product": "Cheese",
              "Id": 12345,
              "SellByDate": "2024-01-11T00:00:00+00:00",
              "IsDelicious": true
            }

Have you tried running it and testing with the UI / Kafka?
If I replace the updated maskRecord with your original one, it works fine. I can debug to the point I can see the masked record returned to RecordRepository but not as easy to see from there what's happening

    public Record maskRecord(Record record) {
        try {
            if(record.getValue().trim().startsWith("{") && record.getValue().trim().endsWith("}")) {
                JsonMaskingFilter foundFilter = null;
                for (JsonMaskingFilter filter : jsonMaskingFilters) {
                    if (record.getTopic().getName().equalsIgnoreCase(filter.getTopic())) {
                        foundFilter = filter;
                    }
                }
                if (foundFilter != null) {
                    return applyMasking(record, foundFilter.getKeys());
                } else {
                    return applyMasking(record, List.of());
                }
            } else {
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record", e);
            return record;
        }
    }

Can you provide your full application.yml, if possible?

akhq:
  connections:
    local:
      properties:
        bootstrap.servers: "localhost:9092"
  security:
    data-masking:
      mode: json_mask_by_default
      json-filters:
        - description: Unmask non-sensitive values
          topic: results
          keys:
            - Product

I'm wondering if it's something to do with the "disable if no filters found" (which I have realised probably shouldnt be in the mask by default mode) nd there being a bug with determining this, or perhaps the JSON parsing

Thanks for the YAML - I'll check it out later.

Just to make this slightly weirder ....

For easier testing, I created a topic and published a single message to it.

When I run the application, the record is returned unmasked. When I run in debug mode and debug through the masker, the record is eventually returned (to the UI) masked as expected ... 😕

@jamfor352
Copy link
Author

Just to make this slightly weirder ....

For easier testing, I created a topic and published a single message to it.

When I run the application, the record is returned unmasked. When I run in debug mode and debug through the masker, the record is eventually returned (to the UI) masked as expected ... 😕

I'm not entirely sure what the issue here is... but I've changed a few things relating to how masking is validated to work (removed the enabled functionality since it doesn't really bring much and breaks the mask by default flow, and switched JSON checking to a different library which may be better suited)

@emerfan
Copy link

emerfan commented Nov 4, 2024

Just to make this slightly weirder ....
For easier testing, I created a topic and published a single message to it.
When I run the application, the record is returned unmasked. When I run in debug mode and debug through the masker, the record is eventually returned (to the UI) masked as expected ... 😕

I'm not entirely sure what the issue here is... but I've changed a few things relating to how masking is validated to work (removed the enabled functionality since it doesn't really bring much and breaks the mask by default flow, and switched JSON checking to a different library which may be better suited)

Does this work as expected for you against a cluster? I'm still seeing the debug vs run issue

@emerfan
Copy link

emerfan commented Nov 4, 2024

Just to make this slightly weirder ....
For easier testing, I created a topic and published a single message to it.
When I run the application, the record is returned unmasked. When I run in debug mode and debug through the masker, the record is eventually returned (to the UI) masked as expected ... 😕

I'm not entirely sure what the issue here is... but I've changed a few things relating to how masking is validated to work (removed the enabled functionality since it doesn't really bring much and breaks the mask by default flow, and switched JSON checking to a different library which may be better suited)

Does this work as expected for you against a cluster? I'm still seeing the debug vs run issue

I've found the issue ... it's with the new isTombstone and isJson methods. The record is initially null when it hits them.

This works:

public interface Masker {

    Logger LOG = LoggerFactory.getLogger(Masker.class);

    Record maskRecord(Record record);

    default boolean isTombstone(Record record) {
        return record.getValue() == null;
    }

   default boolean isJson(Record record) {
        if(isTombstone(record)) {
            return false;
        }
        try {
            new JSONObject(record.getValue());
        } catch (JSONException ex) {
            try {
                new JSONArray(record.getValue());
            } catch (JSONException ex1) {
                return false;
            }
        }
        return true;
    }
}

and

    public Record maskRecord(Record record) {
        try {
            if(isTombstone(record)) {
                LOG.debug("Record at topic {}, partition {}, offset {} is a tombstone, so not masking.", record.getTopic(), record.getPartition(), record.getOffset());
                return record;
            } else if(isJson(record)) {
                return jsonMaskingFilters
                    .stream()
                    .filter(jsonMaskingFilter -> record.getTopic().getName().equalsIgnoreCase(jsonMaskingFilter.getTopic()))
                    .findFirst()
                    .map(filter -> applyMasking(record, filter.getKeys()))
                    .orElseGet(() -> applyMasking(record, List.of()));
            } else {
                LOG.debug("Record at topic {}, partition {}, offset {} is not JSON, so not masking.", record.getTopic(), record.getPartition(), record.getOffset());
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record at topic {}, partition {}, offset {} due to {}", record.getTopic(), record.getPartition(), record.getOffset(), e.getMessage());
            record.setValue("An exception occurred during an attempt to mask this record. This record is unavailable to view due to safety measures from json_mask_by_default to not leak sensitive data. Please contact akhq administrator.");
            return record;
        }
    }

@jamfor352
Copy link
Author

Just to make this slightly weirder ....
For easier testing, I created a topic and published a single message to it.
When I run the application, the record is returned unmasked. When I run in debug mode and debug through the masker, the record is eventually returned (to the UI) masked as expected ... 😕

I'm not entirely sure what the issue here is... but I've changed a few things relating to how masking is validated to work (removed the enabled functionality since it doesn't really bring much and breaks the mask by default flow, and switched JSON checking to a different library which may be better suited)

Does this work as expected for you against a cluster? I'm still seeing the debug vs run issue

I've found the issue ... it's with the new isTombstone and isJson methods. The record is initially null when it hits them.

This works:

public interface Masker {

    Logger LOG = LoggerFactory.getLogger(Masker.class);

    Record maskRecord(Record record);

    default boolean isTombstone(Record record) {
        return record.getValue() == null;
    }

   default boolean isJson(Record record) {
        if(isTombstone(record)) {
            return false;
        }
        try {
            new JSONObject(record.getValue());
        } catch (JSONException ex) {
            try {
                new JSONArray(record.getValue());
            } catch (JSONException ex1) {
                return false;
            }
        }
        return true;
    }
}

and

    public Record maskRecord(Record record) {
        try {
            if(isTombstone(record)) {
                LOG.debug("Record at topic {}, partition {}, offset {} is a tombstone, so not masking.", record.getTopic(), record.getPartition(), record.getOffset());
                return record;
            } else if(isJson(record)) {
                return jsonMaskingFilters
                    .stream()
                    .filter(jsonMaskingFilter -> record.getTopic().getName().equalsIgnoreCase(jsonMaskingFilter.getTopic()))
                    .findFirst()
                    .map(filter -> applyMasking(record, filter.getKeys()))
                    .orElseGet(() -> applyMasking(record, List.of()));
            } else {
                LOG.debug("Record at topic {}, partition {}, offset {} is not JSON, so not masking.", record.getTopic(), record.getPartition(), record.getOffset());
                return record;
            }
        } catch (Exception e) {
            LOG.error("Error masking record at topic {}, partition {}, offset {} due to {}", record.getTopic(), record.getPartition(), record.getOffset(), e.getMessage());
            record.setValue("An exception occurred during an attempt to mask this record. This record is unavailable to view due to safety measures from json_mask_by_default to not leak sensitive data. Please contact akhq administrator.");
            return record;
        }
    }

Good catch! I put this in the domain object as I thought it would be cleaner.
I'll update my branch with this tonight

@jamfor352
Copy link
Author

@emerfan updated my branch, so it should all be working out of the box for you now.

@AlexisSouquiere can I get a re-review please now changes have been applied and bugs fixed? 🙏

@emerfan
Copy link

emerfan commented Nov 5, 2024

@emerfan updated my branch, so it should all be working out of the box for you now.

@AlexisSouquiere can I get a re-review please now changes have been applied and bugs fixed? 🙏

Looks great! I've run it against a substantial cluster. I like the mask all topics by default, and the updated handling of nested objects. This is a great feature, thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
Status: In progress
Development

Successfully merging this pull request may close these issues.

3 participants