-
Notifications
You must be signed in to change notification settings - Fork 1.2k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Data masking: 1. properties & mapping added to ClustersProperties 2. DataMasking provides function that doing masking for specified topic & target 3. Masking policies implemented: MASK, REMOVE, REPLACE
- Loading branch information
Showing
18 changed files
with
847 additions
and
8 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,123 @@ | ||
| # Topics data masking | ||
|
|
||
| You can configure kafka-ui to mask sensitive data shown in Messages page. | ||
|
|
||
| Several masking policies supported: | ||
|
|
||
| ### REMOVE | ||
| For json objects - remove target fields, otherwise - return "null" string. | ||
| ```yaml | ||
| - type: REMOVE | ||
| fields: [ "id", "name" ] | ||
| ... | ||
| ``` | ||
|
|
||
| Apply examples: | ||
| ``` | ||
| { "id": 1234, "name": { "first": "James" }, "age": 30 } | ||
| -> | ||
| { "age": 30 } | ||
| ``` | ||
| ``` | ||
| non-json string -> null | ||
| ``` | ||
|
|
||
| ### REPLACE | ||
| For json objects - replace target field's values with specified replacement string (by default with `***DATA_MASKED***`). Note: if target field's value is object, then replacement applied to all its fields recursively (see example). | ||
|
|
||
| ```yaml | ||
| - type: REPLACE | ||
| fields: [ "id", "name" ] | ||
| replacement: "***" #optional, "***DATA_MASKED***" by default | ||
| ... | ||
| ``` | ||
|
|
||
| Apply examples: | ||
| ``` | ||
| { "id": 1234, "name": { "first": "James", "last": "Bond" }, "age": 30 } | ||
| -> | ||
| { "id": "***", "name": { "first": "***", "last": "***" }, "age": 30 } | ||
| ``` | ||
| ``` | ||
| non-json string -> *** | ||
| ``` | ||
|
|
||
| ### MASK | ||
| Mask target field's values with specified masking characters, recursively (spaces and line separators will be kept as-is). | ||
| `pattern` array specifies what symbols will be used to replace upper-case chars, lower-case chars, digits and other symbols correspondingly. | ||
|
|
||
| ```yaml | ||
| - type: MASK | ||
| fields: [ "id", "name" ] | ||
| pattern: ["A", "a", "N", "_"] # optional, default is ["X", "x", "n", "-"] | ||
| ... | ||
| ``` | ||
|
|
||
| Apply examples: | ||
| ``` | ||
| { "id": 1234, "name": { "first": "James", "last": "Bond!" }, "age": 30 } | ||
| -> | ||
| { "id": "NNNN", "name": { "first": "Aaaaa", "last": "Aaaa_" }, "age": 30 } | ||
| ``` | ||
| ``` | ||
| Some string! -> Aaaa aaaaaa_ | ||
| ``` | ||
|
|
||
| ---- | ||
|
|
||
| For each policy, if `fields` not specified, then policy will be applied to all object's fields or whole string if it is not a json-object. | ||
|
|
||
| You can specify which masks will be applied to topic's keys/values. Multiple policies will be applied if topic matches both policy's patterns. | ||
|
|
||
| Yaml configuration example: | ||
| ```yaml | ||
| kafka: | ||
| clusters: | ||
| - name: ClusterName | ||
| # Other Cluster configuration omitted ... | ||
| masking: | ||
| - type: REMOVE | ||
| fields: [ "id" ] | ||
| topicKeysPattern: "events-with-ids-.*" | ||
| topicValuesPattern: "events-with-ids-.*" | ||
|
|
||
| - type: REPLACE | ||
| fields: [ "companyName", "organizationName" ] | ||
| replacement: "***MASKED_ORG_NAME***" #optional | ||
| topicValuesPattern: "org-events-.*" | ||
|
|
||
| - type: MASK | ||
| fields: [ "name", "surname" ] | ||
| pattern: ["A", "a", "N", "_"] #optional | ||
| topicValuesPattern: "user-states" | ||
|
|
||
| - type: MASK | ||
| topicValuesPattern: "very-secured-topic" | ||
| ``` | ||
| Same configuration in env-vars fashion: | ||
| ``` | ||
| ... | ||
| KAFKA_CLUSTERS_0_MASKING_0_TYPE: REMOVE | ||
| KAFKA_CLUSTERS_0_MASKING_0_FIELDS_0: "id" | ||
| KAFKA_CLUSTERS_0_MASKING_0_TOPICKEYSPATTERN: "events-with-ids-.*" | ||
| KAFKA_CLUSTERS_0_MASKING_0_TOPICVALUESPATTERN: "events-with-ids-.*" | ||
|
|
||
| KAFKA_CLUSTERS_0_MASKING_1_TYPE: REPLACE | ||
| KAFKA_CLUSTERS_0_MASKING_1_FIELDS_0: "companyName" | ||
| KAFKA_CLUSTERS_0_MASKING_1_FIELDS_1: "organizationName" | ||
| KAFKA_CLUSTERS_0_MASKING_1_REPLACEMENT: "***MASKED_ORG_NAME***" | ||
| KAFKA_CLUSTERS_0_MASKING_1_TOPICVALUESPATTERN: "org-events-.*" | ||
|
|
||
| KAFKA_CLUSTERS_0_MASKING_2_TYPE: MASK | ||
| KAFKA_CLUSTERS_0_MASKING_2_FIELDS_0: "name" | ||
| KAFKA_CLUSTERS_0_MASKING_2_FIELDS_1: "surname" | ||
| KAFKA_CLUSTERS_0_MASKING_2_PATTERN_0: 'A' | ||
| KAFKA_CLUSTERS_0_MASKING_2_PATTERN_1: 'a' | ||
| KAFKA_CLUSTERS_0_MASKING_2_PATTERN_2: 'N' | ||
| KAFKA_CLUSTERS_0_MASKING_2_PATTERN_3: '_' | ||
| KAFKA_CLUSTERS_0_MASKING_2_TOPICVALUESPATTERN: "user-states" | ||
|
|
||
| KAFKA_CLUSTERS_0_MASKING_3_TYPE: MASK | ||
| KAFKA_CLUSTERS_0_MASKING_3_TOPICVALUESPATTERN: "very-secured-topic" | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
92 changes: 92 additions & 0 deletions
92
kafka-ui-api/src/main/java/com/provectus/kafka/ui/service/masking/DataMasking.java
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,92 @@ | ||
| package com.provectus.kafka.ui.service.masking; | ||
|
|
||
| import static java.util.stream.Collectors.toList; | ||
|
|
||
| import com.fasterxml.jackson.core.JsonProcessingException; | ||
| import com.fasterxml.jackson.databind.JsonNode; | ||
| import com.fasterxml.jackson.databind.json.JsonMapper; | ||
| import com.fasterxml.jackson.databind.node.ContainerNode; | ||
| import com.google.common.annotations.VisibleForTesting; | ||
| import com.google.common.base.Preconditions; | ||
| import com.provectus.kafka.ui.config.ClustersProperties; | ||
| import com.provectus.kafka.ui.serde.api.Serde; | ||
| import com.provectus.kafka.ui.service.masking.policies.MaskingPolicy; | ||
| import java.util.List; | ||
| import java.util.Optional; | ||
| import java.util.function.UnaryOperator; | ||
| import java.util.regex.Pattern; | ||
| import javax.annotation.Nullable; | ||
| import lombok.Value; | ||
| import org.apache.commons.lang3.StringUtils; | ||
|
|
||
| public class DataMasking { | ||
|
|
||
| private static final JsonMapper JSON_MAPPER = new JsonMapper(); | ||
|
|
||
| @Value | ||
| static class Mask { | ||
| @Nullable | ||
| Pattern topicKeysPattern; | ||
| @Nullable | ||
| Pattern topicValuesPattern; | ||
|
|
||
| MaskingPolicy policy; | ||
|
|
||
| boolean shouldBeApplied(String topic, Serde.Target target) { | ||
| return target == Serde.Target.KEY | ||
| ? topicKeysPattern != null && topicKeysPattern.matcher(topic).matches() | ||
| : topicValuesPattern != null && topicValuesPattern.matcher(topic).matches(); | ||
| } | ||
| } | ||
|
|
||
| private final List<Mask> masks; | ||
|
|
||
| public static DataMasking create(List<ClustersProperties.Masking> config) { | ||
| return new DataMasking( | ||
| config.stream().map(property -> { | ||
| Preconditions.checkNotNull(property.getType(), "masking type not specifed"); | ||
| Preconditions.checkArgument( | ||
| StringUtils.isNotEmpty(property.getTopicKeysPattern()) | ||
| || StringUtils.isNotEmpty(property.getTopicValuesPattern()), | ||
| "topicKeysPattern or topicValuesPattern (or both) should be set for masking policy"); | ||
| return new Mask( | ||
| Optional.ofNullable(property.getTopicKeysPattern()).map(Pattern::compile).orElse(null), | ||
| Optional.ofNullable(property.getTopicValuesPattern()).map(Pattern::compile).orElse(null), | ||
| MaskingPolicy.create(property) | ||
| ); | ||
| }).collect(toList())); | ||
| } | ||
|
|
||
| @VisibleForTesting | ||
| DataMasking(List<Mask> masks) { | ||
| this.masks = masks; | ||
| } | ||
|
|
||
| public UnaryOperator<String> getMaskingFunction(String topic, Serde.Target target) { | ||
| var targetMasks = masks.stream().filter(m -> m.shouldBeApplied(topic, target)).collect(toList()); | ||
| if (targetMasks.isEmpty()) { | ||
| return UnaryOperator.identity(); | ||
| } | ||
| return inputStr -> { | ||
| if (inputStr == null) { | ||
| return null; | ||
| } | ||
| try { | ||
| JsonNode json = JSON_MAPPER.readTree(inputStr); | ||
| if (json.isContainerNode()) { | ||
| for (Mask targetMask : targetMasks) { | ||
| json = targetMask.policy.applyToJsonContainer((ContainerNode<?>) json); | ||
| } | ||
| return json.toString(); | ||
| } | ||
| } catch (JsonProcessingException jsonException) { | ||
| //just ignore | ||
| } | ||
| // if we can't parse input as json or parsed json is not object/array | ||
| // we just apply first found policy | ||
| // (there is no need to apply all of them, because they will just override each other) | ||
| return targetMasks.get(0).policy.applyToString(inputStr); | ||
| }; | ||
| } | ||
|
|
||
| } |
Oops, something went wrong.