Improve and expand client protocol docs #23836
Conversation
mosabua
force-pushed
the
spool
branch
3 times, most recently
from
5cfaa00 to
30d3bfc
Compare
mosabua
force-pushed
the
spool
branch
10 times, most recently
from
57b2ce5 to
c2849c6
Compare
|
@wendigo I have to complete the properties list and the client-protocol page yet - specifically how to configure spooling and aim to do that tomorrow .. but I would appreciate a first look so I can figure out if I am on the right track and how close I am. Also note that the developer guide section should be done later since I dont want to hold this up any longer. |
mosabua
force-pushed
the
spool
branch
3 times, most recently
from
a9e9671 to
edee585
Compare
|
Alright @wendigo .. I think this is good to go now. Let me know if anything else is needed. For example, it might be nice to add config snippets for Azure and GCS but that is optional. |
...spooling-filesystem/src/main/java/io/trino/spooling/filesystem/FileSystemSpoolingConfig.java
Outdated
Show resolved
Hide resolved
mosabua
force-pushed
the
spool
branch
2 times, most recently
from
438567c to
4a82054
Compare
Include the new spooling protocol and its configuration for CLI and JDBC driver.
|
Thanks for all your help @wendigo ! |
| from using the spooling protocol. | ||
| * Requires newer client drivers or client applications that support the spooling | ||
| protocol and actively request usage of the spooling protocol. | ||
| * Clients must have access to the object storage. |
There was a problem hiding this comment.
This mixes two things:
- The protocol does not require clients to have access to object storage. They need to have access to whatever endpoints are referenced by the URLs returned in the responses. These could be object storage, endpoints in the coordinator, workers, etc.
- Object storage is just an implementation strategy for realizing the protocol on the server side. When that implementation is used by the server, then clients need to have access to the object storage endpoints.
There was a problem hiding this comment.
okay .. that makes sense .. how do we write that though so its not completely abstract or super long.
- Clients must have access to the spooled segments. Their locations are specified as
URLs to external object storage, the coordinator, or workers in the cluster.
Thats a lot of hedging.
There was a problem hiding this comment.
Also I can work that into #24288 or another PR .. once we figured out what we want to write.
Description
Revamped client protocol docs and added spooling info.
Additional context and related issues
#22271 and various others.
Release notes
(x) This is not user-visible or is docs only, and no release notes are required.