Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

add runtime config for redacting bytes #4

Merged
merged 4 commits into from
Sep 18, 2020

Conversation

skyzh
Copy link

@skyzh skyzh commented Sep 18, 2020

In this PR, we export a REDACT_BYTES atomic bool globally from protobuf crate. This way, we could redact user data in log without changing too much in TiKV.

After this PR gets merged, we'll continue working on tikv/tikv#8670 to complete the redact log feature.

There's still some room for improvement for this feature, which might be worked on later.

  • configure REDACT_BYTES as a crate feature
  • use some kind of extra info to indicate if we should redact a field, as suggested by @yiwu-arbug
message Region {
    uint64 id = 1;
    bytes start_key = 2 [need_redact = true];
    bytes end_key = 3 [need_redact = true];
}

@skyzh skyzh requested review from nrc, yiwu-arbug and BusyJay September 18, 2020 02:25
@skyzh
Copy link
Author

skyzh commented Sep 18, 2020

BTW, I failed to compile these crates until I changed the dependencies specified in Cargo.toml, like this:

diff --git a/protobuf-codegen-pure-test/Cargo.toml b/protobuf-codegen-pure-test/Cargo.toml
index 5c24777..5b1ff4a 100644
--- a/protobuf-codegen-pure-test/Cargo.toml
+++ b/protobuf-codegen-pure-test/Cargo.toml
@@ -30,4 +30,3 @@ bytes = { version = "0.4", optional = true }
 
 [dependencies.protobuf]
 path = "../protobuf"
-package = "nrc-protobuf"
diff --git a/protobuf-codegen-pure/Cargo.toml b/protobuf-codegen-pure/Cargo.toml
index 4e4736c..bdf294d 100644
--- a/protobuf-codegen-pure/Cargo.toml
+++ b/protobuf-codegen-pure/Cargo.toml
@@ -16,8 +16,8 @@ doctest = false
 bench = false
 
 [dependencies]
-protobuf = { package = "nrc-protobuf", path = "../protobuf", version = "=2.8.0" }
-protobuf-codegen = { package = "nrc-protobuf-codegen", path = "../protobuf-codegen", version = "=2.8.0" }
+protobuf = { path = "../protobuf", version = "=2.8.0" }
+protobuf-codegen = { path = "../protobuf-codegen", version = "=2.8.0" }
 
 [[bin]]
 
diff --git a/protobuf-codegen/Cargo.toml b/protobuf-codegen/Cargo.toml
index 3b7aa02..9cf6903 100644
--- a/protobuf-codegen/Cargo.toml
+++ b/protobuf-codegen/Cargo.toml
@@ -19,7 +19,7 @@ A minor fork of https://github.com/stepancheg/rust-protobuf/
 bench = false
 
 [dependencies]
-protobuf = { package = "protobuf", version = "=2.8.0" }
+protobuf = { path="../protobuf" }
 heck = "0.3"
 
 [[bin]]
diff --git a/protobuf-codegen/src/bin/protobuf-bin-gen-rust-do-not-use.rs b/protobuf-codegen/src/bin/protobuf-bin-gen-rust-do-not-use.rs
index 25faa4a..fc66d27 100644
--- a/protobuf-codegen/src/bin/protobuf-bin-gen-rust-do-not-use.rs
+++ b/protobuf-codegen/src/bin/protobuf-bin-gen-rust-do-not-use.rs
@@ -1,5 +1,5 @@
 extern crate protobuf;
-extern crate nrc_protobuf_codegen as protobuf_codegen;
+extern crate protobuf_codegen;
 
 use std::fs::*;
 use std::io::Read;
diff --git a/protobuf-codegen/src/bin/protoc-gen-rust.rs b/protobuf-codegen/src/bin/protoc-gen-rust.rs
index f7509d8..97ac7d2 100644
--- a/protobuf-codegen/src/bin/protoc-gen-rust.rs
+++ b/protobuf-codegen/src/bin/protoc-gen-rust.rs
@@ -1,4 +1,4 @@
-extern crate nrc_protobuf_codegen as protobuf_codegen;
+extern crate protobuf_codegen;
 
 fn main() {
     protobuf_codegen::protoc_gen_rust_main();
diff --git a/protobuf-test-common/Cargo.toml b/protobuf-test-common/Cargo.toml
index 10141ce..1491b9d 100644
--- a/protobuf-test-common/Cargo.toml
+++ b/protobuf-test-common/Cargo.toml
@@ -34,8 +34,7 @@ lazy_static = "1.1.*"
 
 [dependencies.protobuf]
 path = "../protobuf"
-package = "nrc-protobuf"
 
 [dependencies.protobuf-codegen]
 path = "../protobuf-codegen"
-package = "nrc-protobuf-codegen"
+
diff --git a/protobuf-test/Cargo.toml b/protobuf-test/Cargo.toml
index c5c25f3..6a81711 100644
--- a/protobuf-test/Cargo.toml
+++ b/protobuf-test/Cargo.toml
@@ -30,5 +30,4 @@ serde_json   = { version = "1.0", optional = true }
 bytes = { version = "0.4", optional = true }
 
 [dependencies.protobuf]
-path = "../protobuf"
-package = "nrc-protobuf"
+path = "../protobuf"
\ No newline at end of file
diff --git a/protoc-rust/Cargo.toml b/protoc-rust/Cargo.toml
index ad61a50..e8ff6b0 100644
--- a/protoc-rust/Cargo.toml
+++ b/protoc-rust/Cargo.toml
@@ -14,9 +14,9 @@ doctest = false
 bench = false
 
 [dependencies]
-protoc = { path = "../protoc", version = "=2.8.0" }
-protobuf = { package = "nrc-protobuf", path = "../protobuf", version = "=2.8.0" }
-protobuf-codegen = { package = "nrc-protobuf-codegen", path = "../protobuf-codegen", version = "=2.8.0" }
+protoc = { path = "../protoc" }
+protobuf = { path = "../protobuf" }
+protobuf-codegen = { path = "../protobuf-codegen" }
 tempfile = "3"
 
 [package.metadata.docs.rs]
diff --git a/protoc/test-protoc/Cargo.toml b/protoc/test-protoc/Cargo.toml
index 1e36cad..f23a277 100644
--- a/protoc/test-protoc/Cargo.toml
+++ b/protoc/test-protoc/Cargo.toml
@@ -10,7 +10,7 @@ doctest = false
 bench = false
 
 [dependencies]
-protobuf = { package = "nrc-protobuf", path = "../../protobuf" }
+protobuf = { path = "../../protobuf" }
 
 [features]
 default-features = []

These changes are not included in this PR, but seems necessary to make it compile.

protobuf/src/core.rs Outdated Show resolved Hide resolved
protobuf/src/core.rs Outdated Show resolved Hide resolved
protobuf/src/lib.rs Outdated Show resolved Hide resolved
Copy link

@yiwu-arbug yiwu-arbug left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants