Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Row content being logged as part of CDC ErrMessageTooLarge logging #9031

Closed
darraes opened this issue May 23, 2023 · 6 comments · Fixed by #9032
Closed

Row content being logged as part of CDC ErrMessageTooLarge logging #9031

darraes opened this issue May 23, 2023 · 6 comments · Fixed by #9032
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.

Comments

@darraes
Copy link
Contributor

darraes commented May 23, 2023

What did you do?

If a message is too large, both the Canal JSON and the Open Protocol encoders are logging the full message.

You can see the issue in 3 files:
canal_json_row_event_encoder.go:347
canal_json_tx_event_encoder.go:64
canal_json_row_event_encoder.go:69

What did you expect to see?

The message itself (the row data) should not be printed in logs as this is a PII violation

What did you see instead?

The content of the row

Versions of the cluster

Upstream TiDB cluster version (execute SELECT tidb_version(); in a MySQL client):

(paste TiDB cluster version here)

Upstream TiKV version (execute tikv-server --version):

(paste TiKV version here)

TiCDC version (execute cdc version):

v7.0.0
@darraes darraes added area/ticdc Issues or PRs related to TiCDC. type/bug The issue is confirmed as a bug. labels May 23, 2023
@darraes darraes mentioned this issue May 23, 2023
Merged
@darraes
Copy link
Contributor Author

darraes commented May 23, 2023

/area ticdc

@darraes
Copy link
Contributor Author

darraes commented May 23, 2023

/severity major

@charleszheng44
Copy link
Contributor

/severity minor

@nongfushanquan
Copy link
Contributor

/remove-label may-affects-7.1
/remove-label may-affects-6.5
/label affects-6.5
/label affects-7.1

@ti-chi-bot ti-chi-bot bot added affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. and removed may-affects-7.1 may-affects-6.5 labels May 25, 2023
@nongfushanquan
Copy link
Contributor

/remove-label may-affects-6.1
/label affects-6.1

@ti-chi-bot ti-chi-bot bot added affects-6.1 This bug affects the 6.1.x(LTS) versions. and removed may-affects-6.1 labels May 25, 2023
@ti-chi-bot ti-chi-bot bot closed this as completed in #9032 Jun 1, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jun 1, 2023
@darraes
codec(cdc) Removing logging of message bodies to avoid PII issues (#9032
156d1d5 
)

close #9031
This was referenced Jun 1, 2023
Open
Merged
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jun 2, 2023
@ti-chi-bot
codec(cdc) Removing logging of message bodies to avoid PII issues (#9032
2a2f55c 
) (#9115)

close #9031
@nongfushanquan nongfushanquan mentioned this issue Jun 9, 2023
Merged
17 tasks
@nongfushanquan
Copy link
Contributor

/remove-label affects-6.1

@ti-chi-bot ti-chi-bot bot removed the affects-6.1 This bug affects the 6.1.x(LTS) versions. label Jun 26, 2023
ti-chi-bot bot pushed a commit that referenced this issue Jul 4, 2023
@ti-chi-bot
codec(cdc) Removing logging of message bodies to avoid PII issues (#9032
036ca3e 
) (#9116)

close #9031
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
affects-6.5 This bug affects the 6.5.x(LTS) versions. affects-7.1 This bug affects the 7.1.x(LTS) versions. area/ticdc Issues or PRs related to TiCDC. severity/minor type/bug The issue is confirmed as a bug.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

codec(cdc) Removing logging of message bodies to avoid PII issues darraes/tiflow
4 participants
@darraes @charleszheng44 @asddongmen @nongfushanquan