Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Implement parallel read from S3 for exchange storage #11174

Merged
merged 5 commits into from
Mar 16, 2022
Merged

Implement parallel read from S3 for exchange storage #11174

merged 5 commits into from
Mar 16, 2022

Conversation

linzebing
Copy link
Member

@linzebing linzebing commented Feb 23, 2022 •
edited
Loading

Description

Is this change a fix, improvement, new feature, refactoring, or other?

Improvement of the read path for exchange sources.

Is this a change to the core query engine, a connector, client library, or the SPI interfaces? (be specific)

To the trino-exchange plugin.

How would you describe this change to a non-technical end user or system administrator?

This PR implements parallel read for s3-based exchange storage, including inter-file and intra-file level parallelism, and improves the overall throughput of fault tolerant execution.

Related issues, pull requests, and links

#11050

Documentation

(x) No documentation is needed.
( ) Sufficient documentation is included in this PR.
( ) Documentation PR is available with #prnumber.
( ) Documentation issue #issuenumber is filed, and can be handled later.

After implementation/configs stabilize and get well tested, we will add documentations related to fault tolerant execution.

Release notes

() No release notes entries required.
(x) Release notes entries required with the following suggested text:

# Exchange plugin
* Implement parallel read for ExchangeSource. In particular, for S3-based spooling, implemented inter and intra file parallel read. ({issue}`11050`)

@cla-bot cla-bot bot added the cla-signed label Feb 23, 2022
arhimondr
arhimondr reviewed Feb 24, 2022
View reviewed changes
Copy link
Contributor

@arhimondr arhimondr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great job! Some comments

plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeConfig.java Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeConfig.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeConfig.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeManager.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSink.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
@linzebing linzebing mentioned this pull request Feb 24, 2022
Merged
losipiuk
losipiuk reviewed Feb 25, 2022
View reviewed changes
Copy link
Member

@losipiuk losipiuk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice :)
I put some comments - but all of that are editorials.

plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeConfig.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeManager.java Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/ExchangeSourceFile.java Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/ExchangeSourceFile.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
...-exchange/src/main/java/io/trino/plugin/exchange/s3/BufferWriteAsyncResponseTransformer.java Show resolved Hide resolved
...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Outdated Show resolved Hide resolved
...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Outdated
if (oldBuffer != null) {
verify(bufferPosition <= bufferFill,
"bufferPosition (%s) expected to be smaller or equal to bufferFill (%s)", bufferPosition, bufferFill);
int rest = bufferFill - bufferPosition;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

remainderSize?

...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Outdated
}

// Make sure S3 read request byte ranges align with part sizes for best performance
int readableParts = toIntExact(buffer.length - bufferFill) / partSize;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

readableParts was a bit confusing to me as it felt it characterizes file, while it actually characterizes buffer.
Maybe rename to bufferPartsCapacity or sth similar?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It actually characterizes file. So I feel readablePart is quite readable here.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why file? It says how many parts can we fit in a buffer, right?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's both. Imagine a file of 12MB, with part size 5MB, then it will be stored in three parts: 5MB-5MB-2MB

...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Show resolved Hide resolved
@linzebing linzebing force-pushed the parallel-read-v2 branch 3 times, most recently from 2741267 to f164809 Compare February 28, 2022 22:41
@linzebing linzebing mentioned this pull request Mar 1, 2022
Merged
@linzebing linzebing force-pushed the parallel-read-v2 branch 2 times, most recently from 11fa33e to f2d6621 Compare March 1, 2022 01:47
@linzebing linzebing requested a review from arhimondr March 1, 2022 01:48
@losipiuk
Copy link
Member

losipiuk commented Mar 1, 2022

CI: raptor #11252

arhimondr
arhimondr approved these changes Mar 1, 2022
View reviewed changes
Copy link
Contributor

@arhimondr arhimondr left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM % nits

plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Outdated Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeSource.java Show resolved Hide resolved
plugin/trino-exchange/src/main/java/io/trino/plugin/exchange/FileSystemExchangeStorage.java Outdated Show resolved Hide resolved
...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Outdated Show resolved Hide resolved
...in/trino-exchange/src/main/java/io/trino/plugin/exchange/s3/S3FileSystemExchangeStorage.java Show resolved Hide resolved
@linzebing linzebing force-pushed the parallel-read-v2 branch 2 times, most recently from 7dbe584 to faf4a1c Compare March 3, 2022 22:39
@linzebing linzebing force-pushed the parallel-read-v2 branch 3 times, most recently from 88c99b0 to be60aec Compare March 9, 2022 01:07
@linzebing
Copy link
Member Author

linzebing commented Mar 9, 2022 •
edited
Loading

Test failure due to #11368

@arhimondr arhimondr merged commit f4f1496 into trinodb:master Mar 16, 2022
@github-actions github-actions bot added this to the 374 milestone Mar 16, 2022
@mosabua mosabua mentioned this pull request Mar 16, 2022
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@arhimondr arhimondr arhimondr approved these changes

@losipiuk losipiuk losipiuk approved these changes

Assignees
No one assigned
Labels
cla-signed
Milestone
374
Development

Successfully merging this pull request may close these issues.
3 participants
@linzebing @losipiuk @arhimondr