Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Optimize memory usage of RowChangedEvent #10386

Closed
lidezhu opened this issue Dec 28, 2023 · 4 comments · Fixed by #10483
Closed

Optimize memory usage of RowChangedEvent #10386

lidezhu opened this issue Dec 28, 2023 · 4 comments · Fixed by #10483
Assignees
@lidezhu
Labels
area/ticdc Issues or PRs related to TiCDC. type/enhancement The issue or PR belongs to an enhancement.

Comments

@lidezhu
Copy link
Collaborator

lidezhu commented Dec 28, 2023 •
edited
Loading

Description

RowChangedEvent represents a row changed event and contains required data and metadata in it. But each of them copy some schema information into it(like type, name) which may cause unnecessary memory usage.
And the memory profile on some real env shows that the decoding of RowChangedEvent is the main cause of memory consumption.

Enhancement

Avoid copy schema info into RowChangedEvent.
@lidezhu
Copy link
Collaborator Author

lidezhu commented Dec 28, 2023

/label type/enhancement

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 28, 2023

@lidezhu: The label(s) type/enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, ok-to-test, needs-ok-to-test, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, question, release-blocker, wontfix, MariaDB.

In response to this:

/label type/enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@lidezhu
Copy link
Collaborator Author

lidezhu commented Dec 28, 2023

/label enhancement

@ti-chi-bot ti-chi-bot
Copy link
Contributor

ti-chi-bot bot commented Dec 28, 2023

@lidezhu: The label(s) enhancement cannot be applied. These labels are supported: duplicate, bug-from-internal-test, bug-from-user, ok-to-test, needs-ok-to-test, affects-5.4, affects-6.1, affects-6.5, affects-7.1, affects-7.5, may-affects-5.4, may-affects-6.1, may-affects-6.5, may-affects-7.1, may-affects-7.5, needs-cherry-pick-release-5.4, needs-cherry-pick-release-6.1, needs-cherry-pick-release-6.5, needs-cherry-pick-release-7.1, needs-cherry-pick-release-7.5, question, release-blocker, wontfix, MariaDB.

In response to this:

/label enhancement

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

@lidezhu lidezhu changed the title Optimize memory usage of TiCDC Optimize memory usage of RowChangedEvent Dec 28, 2023
@asddongmen asddongmen added type/enhancement The issue or PR belongs to an enhancement. area/ticdc Issues or PRs related to TiCDC. labels Dec 29, 2023
@zhangjinpeng87 zhangjinpeng87 mentioned this issue Dec 29, 2023
Open
20 tasks
@lidezhu lidezhu mentioned this issue Jan 18, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jan 19, 2024
@lidezhu
cdc: fix build table info (#10492)
33436f9 
ref #10386
@lidezhu lidezhu mentioned this issue Jan 22, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Jan 24, 2024
@lidezhu
cdc: remove some unnecessary fields from RowChangedEvent (#10517)
3e4522d 
ref #10386
@lidezhu lidezhu mentioned this issue Feb 1, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 20, 2024
@lidezhu
*(ticdc): optimize memory usage of RowChangedEvent (#10483)
a8c7563 
close #10386
@lidezhu lidezhu mentioned this issue Feb 21, 2024
Merged
ti-chi-bot bot pushed a commit that referenced this issue Feb 21, 2024
@lidezhu
*: fix dm integration test failure (#10631)
ac76591 
ref #10386
3AceShowHand pushed a commit to 3AceShowHand/tiflow that referenced this issue Feb 22, 2024
@lidezhu @3AceShowHand
*: fix dm integration test failure (pingcap#10631)
fa615c6 
ref pingcap#10386
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@lidezhu lidezhu

Labels
area/ticdc Issues or PRs related to TiCDC. type/enhancement The issue or PR belongs to an enhancement.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

*(ticdc): optimize memory usage of RowChangedEvent lidezhu/tiflow
2 participants
@asddongmen @lidezhu