Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Log redact on TiFlash #1277

Closed
6 of 7 tasks
JaySon-Huang opened this issue Dec 10, 2020 · 5 comments
Closed
6 of 7 tasks

Log redact on TiFlash #1277

JaySon-Huang opened this issue Dec 10, 2020 · 5 comments
Assignees

Comments

@JaySon-Huang
Copy link
Contributor

JaySon-Huang commented Dec 10, 2020

Log redacted: Avoid printing user data (mostly user keys) to info log. Replace the value into ?.
Spec

Logs to be redacted:

Related issue in TiDB: pingcap/tidb#18566
Related PR in TiKV: tikv/tikv#8670

@solotzg
Copy link
Contributor

solotzg commented Dec 10, 2020

ClusterManager only handle range of table record like [t tid _r, t tid+1 ), no user key will be printed in log.
Proxy has already contains related modify about redact-log, which could be enable with config [security] redact-info-log.

@JaySon-Huang
Copy link
Contributor Author

JaySon-Huang commented Dec 14, 2020

For Protobuf struct in logging

Protobuf provides a class TextFormat::Printer for formatting. We can reuse some code of that class and hide the value of fields with option [need_redact = true] like this PR: pingcap/rust-protobuf#4

https://github.com/protocolbuffers/protobuf/blob/fdc35840b95f56c3aef44a3fc6ae5991f21620a5/src/google/protobuf/text_format.h#L242-L244

https://github.com/protocolbuffers/protobuf/blob/bd9a7104e11740e4bcfbde46c190c2d908ef331a/src/google/protobuf/text_format.cc#L86-L111

Related article on how to use protobuf reflection: https://zhuanlan.zhihu.com/p/313369306

@JaySon-Huang
Copy link
Contributor Author

TODO: Update document in this URL https://docs.pingcap.com/tidb/stable/log-redaction

@JaySon-Huang
Copy link
Contributor Author

TODO: Update document in this URL https://docs.pingcap.com/tidb/stable/log-redaction

The PRs to fix those documents:
pingcap/docs-cn#5687
pingcap/docs-cn#5688

@JaySon-Huang JaySon-Huang changed the title Log redact in TiFlash Log redact on TiFlash Mar 9, 2021
@JaySon-Huang
Copy link
Contributor Author

close as the Protobuf struct in logging is not print to logging because the default level is INFO

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants