Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Pipe/Subscription: Filter out non-working DR PipeTasks in CN & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor #14059

Merged
merged 32 commits into from
Nov 26, 2024

Conversation

luoluoyuyu
Copy link
Contributor

@luoluoyuyu luoluoyuyu commented Nov 12, 2024

Description

  1. Why does the RPC request not decrease?
    Because Pipe cannot detect the modification of the DataBase name, the modification of the DataBase name will bring about the modification of the scope of Pipe operation
  2. Why does it need to be deeply copied every time a request is sent?
    Because the PipeInfo in CN cannot be modified, PipeInfo is the last guarantee of Pipe synchronization

This PR has:

  • been self-reviewed.
    • concurrent read
    • concurrent write
    • concurrent read and write
  • added documentation for new or modified features or behaviors.
  • added Javadocs for most classes and all non-trivial methods.
  • added or updated version, license, or notice information
  • added comments explaining the "why" and the intent of the code wherever would not be obvious
    for an unfamiliar reader.
  • added unit tests or modified existing tests to cover new code paths, ensuring the threshold
    for code coverage.
  • added integration tests.
  • been tested in a test IoTDB cluster.

Key changed/added classes (or packages if there are too many classes) in this PR

@luoluoyuyu luoluoyuyu marked this pull request as draft November 12, 2024 08:42
@SteveYurongSu SteveYurongSu self-assigned this Nov 13, 2024
@luoluoyuyu luoluoyuyu marked this pull request as ready for review November 13, 2024 02:51
@luoluoyuyu luoluoyuyu changed the title Pipe: Reduce the number of Pipes started in non-valid range DataRegion Pipe: DN's Pipe runs tasks to prune and fix invalid parsing of captured table models TSFile Nov 20, 2024
@luoluoyuyu luoluoyuyu changed the title Pipe: DN's Pipe runs tasks to prune and fix invalid parsing of captured table models TSFile Pipe: Reduce the number of Pipes started in non-valid range DataRegion Nov 20, 2024
@SteveYurongSu SteveYurongSu changed the title Pipe: Reduce the number of Pipes started in non-valid range DataRegion Pipe: Reduce the number of pipe tasks started in non-valid DataRegions & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Nov 25, 2024
@SteveYurongSu SteveYurongSu changed the title Pipe: Reduce the number of pipe tasks started in non-valid DataRegions & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Pipe: Reduce the number of pipe tasks started in non-valid DataRegions from procedure ctors & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Nov 25, 2024
@SteveYurongSu SteveYurongSu changed the title Pipe: Reduce the number of pipe tasks started in non-valid DataRegions from procedure ctors & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Pipe: Filter out non-working DR PipeTasks in CN & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Nov 25, 2024
@SteveYurongSu SteveYurongSu changed the title Pipe: Filter out non-working DR PipeTasks in CN & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Pipe/Subscription: Filter out non-working DR PipeTasks in CN & Reduce model judgement cost in PipeHistoricalDataRegionTsFileAndDeletionExtractor Nov 25, 2024
@SteveYurongSu SteveYurongSu merged commit 68daea6 into apache:master Nov 26, 2024
31 of 32 checks passed
@luoluoyuyu luoluoyuyu deleted the reduce-invalid-pipe-startup branch November 29, 2024 10:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants