-
Notifications
You must be signed in to change notification settings - Fork 3.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support concurrent export of query results #6539
Conversation
fe/fe-core/src/main/java/org/apache/doris/qe/SessionVariable.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/analysis/OutFileClause.java
Outdated
Show resolved
Hide resolved
fe/fe-core/src/main/java/org/apache/doris/planner/ExchangeNode.java
Outdated
Show resolved
Hide resolved
} else { | ||
_storage_type = TStorageBackendType::BROKER; | ||
} | ||
_fragment_instance_id.hi = 12345678987654321; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
What's this?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
In order to be compatible with the old version of Fe and the new version of be
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Add comment to explain.
BTW, I suggest to add FileNamePrefix
column in the result of outfile operation. For easy to get full name of exported files. Such as:
+------------+-----------+-----------+-------------+------------------------------------------+
| FileNumber | TotalRows | FileSize | URL |FileNamePrefix |
+------------+-----------+-----------+-------------+------------------------------------------+
| 1 | 123605 | 361061014 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10325|
| 1 | 128180 | 374334318 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10326|
| 1 | 125156 | 365569023 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10327|
| 1 | 124096 | 362395588 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10328|
| 1 | 124862 | 364727515 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10329|
| 1 | 124520 | 363649600 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10330|
| 1 | 124447 | 363479285 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10331|
| 1 | 125134 | 365490611 | 127.0.0.1 |my_file2_46e9ef9b66924a79-92f887d43be10332|
+------------+-----------+-----------+-------------+------------------------------------------+
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
It's not easy to add, there will be compatibility issues
This pr mainly supports 1. Export query result sets concurrently 2. Query result set export supports s3 protocol Among them, there are several preconditions for concurrently exporting query result sets 1. Enable concurrent export variables 2. The query itself can be exported concurrently (some queries containing sort nodes at the top level cannot be exported concurrently) 3. Export the s3 protocol used instead of the broker After exporting the result set concurrently, the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}
3203362
to
29e5e79
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
LGTM
PR approved by at least one committer and no changes requested. |
PR approved by anyone and no changes requested. |
PR approved by at least one committer and no changes requested. |
legacy planner impl PR: apache#6539
legacy planner impl PR: apache#6539
legacy planner impl PR: apache#6539
legacy planner impl PR: #6539
legacy planner impl PR: #6539
Proposed changes
This pr mainly supports
Among them, there are several preconditions for concurrently exporting query result sets
(some queries containing sort nodes at the top level cannot be exported concurrently)
After exporting the result set concurrently,
the file prefix is changed to outfile_{query_instance_id}_filenumber.{file_format}
Types of changes
What types of changes does your code introduce to Doris?
Put an
x
in the boxes that applyChecklist
Put an
x
in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code.