Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Allow app to accept/reject/retry before handshake begins #1752

Merged
merged 7 commits into from
Apr 3, 2024

Conversation

gretchenfrage
Copy link
Contributor

@gretchenfrage gretchenfrage commented Jan 30, 2024

Closes #1661

Please see individual commit messages.

@Ralith
Copy link
Collaborator

Ralith commented Jan 31, 2024

Thank you for working on this! I've wanted this feature for a long time, and I'm excited about the amount of discretion it gives applications for practically no increase in intrinsic API complexity.

It may be a few days before I have time to dig into reviewing this properly, but I can see it's turned out to be a fairly large change. Any chance you could break it up into a series of smaller incremental, self-contained commits?

Copy link
Member

@djc djc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I skimmed your changes to try and give some high-level feedback.

In particular, the changes to the -proto Endpoint are just too hard to review as one commit. Suggest you refactor into an API that better fits your proposal before making the final change to split off the API that allows the caller to make a decision on handling incoming connections.

quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn/examples/client.rs Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/endpoint.rs Outdated Show resolved Hide resolved
@gretchenfrage
Copy link
Contributor Author

Thanks for the review! Splitting this into more incremental commits and making the other changes you mentioned makes sense to me. Just so you know, though, it might be a little bit until I can dedicate more time to this--but I'll get back to you when it's ready to be looked at again.

@gretchenfrage
Copy link
Contributor Author

Should be ready for re-review now. I split this up into various commits which should make this a lot easier to see what I'm doing if you go through them one at a time, and made some of the other changes you requested too.

Why not expose ConnectingState directly?

I don't really see how this would work--at least, I'm not sure how to reconcile that idea with Ralith's request that this functionality be placed into the existing Connecting API. Which, although that wasn't my original proposal, I do like the idea of. If you have something in mind though I'd be happy to talk about it.

... because this gets pretty messy.

Yeah that's fair--I've refactored that area in general, it should be a lot cleaner now.

quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/connection.rs Outdated Show resolved Hide resolved
quinn/src/tests.rs Show resolved Hide resolved
quinn/src/tests.rs Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
@gretchenfrage
Copy link
Contributor Author

Should we introduce a new public future (perhaps Handshaking) returned by accept, and move this and into_0rtt there?

Pros:

  • Reduced side effects here and in into_0rtt
  • No need for an AcceptError in into_0rtt
  • Less risk of panics in is_validated/may_retry/reject/retry

Cons:

  • Duplication of local_ip/remote_address

Seems like a win to me. We could even get rid of the state enum entirely by implementing IntoFuture rather than Future to support direct await calls.

Alright, I've done a considerable refactor which I think achieves a good compromise between the various desires here, and results in a smaller PR previously. I'm no longer introducing different internal states to Connecting--in fact quinn::Connecting is completely untouched.

Awaiting accept on a quinn::Endpoint now returns a new quinn::IncomingConnection type. quinn::IncomingConnection has the following methods:

  • All the methods quinn_proto::IncomingConnection has
  • fn accept(self) -> Result<Connecting, ConnectionError>
  • fn reject(self)
  • fn retry(self) -> Result<(), RetryError>
    • (new quinn::RetryError is an error but also is just a wrapper around the quinn::IncomingConnection)
  • fn ignore(self)

Furthermore, it implements IntoFuture<Output=Result<Connection, ConnectionError>>, the output type being the same as Connecting. This means that code which was just doing some basic endpoint.accept().await.unwrap().await (or similar) logic will still work without modification.

Copy link
Collaborator

@Ralith Ralith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, this is looking really nice!

quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/tests/mod.rs Outdated Show resolved Hide resolved
quinn/src/incoming_connection.rs Outdated Show resolved Hide resolved
quinn/src/endpoint.rs Outdated Show resolved Hide resolved
quinn/src/incoming_connection.rs Outdated Show resolved Hide resolved
quinn/src/incoming_connection.rs Outdated Show resolved Hide resolved
quinn/src/tests.rs Show resolved Hide resolved
@Ralith
Copy link
Collaborator

Ralith commented Feb 14, 2024

I'm not sure how to feel about having both Endpoint::accept and IncomingConnection::accept which do rather different things. On the one hand, it's clear enough in context on each type, but in an abstract sense having to accept each connection twice is funny and might cause confusion in informal discussion: "Where did you see that error?" "After I accepted a connection.". Would IncomingConnection::connect be a clearer name? But that wouldn't match the proto layer...

@Ralith
Copy link
Collaborator

Ralith commented Feb 14, 2024

After sleeping on it I think the current naming is probably the sweet spot.

@gretchenfrage
Copy link
Contributor Author

After sleeping on it I think the current naming is probably the sweet spot.

Huh alright. I mean I'd be fine renaming it to IncomingConnection::connect on both the quinn and quinn-proto layer but yeah leaving as-is works.

@gretchenfrage
Copy link
Contributor Author

Could we simplify the API by returning Connecting unconditionally, but constructing it such that it immediately yields a ConnectionError when awaited if the proto-layer accept failed? That seems like it'd reduce the number of distinct error paths in downstream code, and reduce churn. For example, the change to the accept loop in quinn/examples/server.rs, which introduces a regression where accept errors aren't logged, would be avoided.

On second thought, that's just .await again. No harm in being a little more fine-grained with the explicit API, I suppose!

If it helps clarify the situation: when changing server applications, I suspect what might end up being done most naturally in most cases is just changing functions which accept a Connecting to instead accept an IncomingConnection but then awaiting it all the same. See for example the entire diff to perf_server.rs:

diff --git a/perf/src/bin/perf_server.rs b/perf/src/bin/perf_server.rs
index 169ca65e..4b710268 100644
--- a/perf/src/bin/perf_server.rs
+++ b/perf/src/bin/perf_server.rs
@@ -123,7 +123,7 @@ async fn run(opt: Opt) -> Result<()> {
     Ok(())
 }
 
-async fn handle(handshake: quinn::Connecting, opt: Arc<Opt>) -> Result<()> {
+async fn handle(handshake: quinn::IncomingConnection, opt: Arc<Opt>) -> Result<()> {
     let connection = handshake.await.context("handshake failed")?;
     debug!("{} connected", connection.remote_address());
     tokio::try_join!(

Thus it's the new IncomingConnectionFuture struct that implicitly does the work of acting as a little adapter which can be used as if it were "returning Connecting unconditionally, but constructing it such that it immediately yields a ConnectionError when awaited if the proto-layer accept failed"

@gretchenfrage
Copy link
Contributor Author

I re-added the RetryPolicy callback to the quinn-proto testing utilities, due to me re-added the quinn-proto reject_remote_address test (now renamed to reject_manually). With regards to your previous comment on this topic:

Could we reduce complexity by just taking an IncomingConnectionResponse directly here? No downstream code seems to need arbitrary logic.

It would not actually be that simple, because RetryPolicy::yes() is not as simple as unconditionally returning IncomingConnectionResponse::Accept--it only does so if the IncomingConnection is not validated.

@gretchenfrage
Copy link
Contributor Author

All comments should be resolved, and I just refactored the commit history and force-pushed again so it should be more clean now.

Copy link
Collaborator

@Ralith Ralith left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM aside from a few last minor issues.

quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/tests/util.rs Outdated Show resolved Hide resolved
quinn/src/endpoint.rs Outdated Show resolved Hide resolved
quinn/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/endpoint.rs Outdated Show resolved Hide resolved
quinn/src/lib.rs Outdated Show resolved Hide resolved
@gretchenfrage
Copy link
Contributor Author

gretchenfrage commented Mar 29, 2024

There were some non-trivialities to removing concurrent_connections. I took the liberty of refactoring some things a tiny bit. Let me know if you disagree with these decisions.

Updated branch to remove concurrent_connections from ServerConfig. Adds open_connections(&self) -> usize method to Endpoint which user can use to enforce concurrent connection limit manually. Also, now removes existing reject_new_connections method on endpoint, as that relies on concurrent_connections and also can now just be done by user manually with the new "incoming" API we're adding.

Previous version of this branch added ConnectionError::ConnectionLimitExceeded variant. New version instead adds ConnectionError::CidsExhausted variant. In previous version of this branch, check_connection_limit method was factored out and called when accepting connections. It did two things: check concurrent connections, and check is_full, and cause ConnectionLimitExceeded error if either case true. In new version of this branch, we rename is_full to cids_exhausted and just directly call it when accepting connections and cause CidsExhausted error if true.

On main, ConnectError has TooManyConnections variant, with comment "The number of active connections on the local endpoint is at the limit." It is not triggered if concurrent_connections limit is reached (which I think is confusing with respect to the error name and message), but rather is only triggered if is_full returns true. We rename ConnectError::TooManyConnections to ConnectError::CidsExhausted so it looks basically the same as ConnectionError::CidsExhausted.


Other changes I made to this branch:

  • Squashed your quinn: improve TransmitState abstraction into prev
  • Adds quinn proto AcceptError as noted in previous comment
  • Adds --connection-limit parameter to server example

quinn/src/incoming.rs Outdated Show resolved Hide resolved
lijunwangs added a commit to lijunwangs/solana that referenced this pull request Mar 31, 2024
lijunwangs added a commit to lijunwangs/jito-relayer that referenced this pull request Mar 31, 2024
@Ralith
Copy link
Collaborator

Ralith commented Mar 31, 2024

I went through and s/reject/refuse/ in the API for consistency w/ the spec and general IETF convention, now that we no longer need to worry about not matching fn reject_new_connections.

@lijunwangs
Copy link
Contributor

What are the remaining outstanding issues? Can we get this merged into main if no more issues? @djc @Ralith

Copy link
Member

@djc djc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a few more minor questions/comments, but I think we should merge this soon.

quinn/src/endpoint.rs Outdated Show resolved Hide resolved
quinn-proto/src/tests/util.rs Outdated Show resolved Hide resolved
quinn/src/endpoint.rs Show resolved Hide resolved
gretchenfrage and others added 7 commits April 2, 2024 17:46
Subsequent commits will allow the user to achieve the removed behavior
manually and more flexibly.

- Removes concurrent_connections from ServerConfig, as well as
  corresponding check in early_validate_first_packet.
- Adds ConnectionError::CidsExhausted error, although it is not yet
  instantiated.
- Renames ConnectError variant TooManyConnections to CidsExhausted.
- Renames proto Endpoint internal method is_full to cids_exhausted.
- Adds method open_connections to Endpoint (both proto and quinn).
- Removes Endpoint method reject_new_connection from Endpoint (both
  proto and quinn).
- Deletes obselete tests concurrent_connections_full and
  reject_new_connections.
This commit factors out the two fields of a quinn::Endpoint's State
necessary to process a proto::Transmit into a new sub-struct,
TransmitState. This is to alleviate borrowing issues, because
proto::Transmit will soon be called from more call sites than
previously.

The bulk of this code change is just moving around existing code.

Co-authored-by: Dirkjan Ochtman <dirkjan@ochtman.nl>
This commit refactors the logic for a quinn_proto::Endpoint accepting
an incoming connection so that it constructs an explicit Incoming struct
containing all the necessary state to accept/reject/retry the
connection-creating packet. However, the external API stays the same.

The bulk of this code change is just moving around existing code.
Additionally, adds some gitignore lines I was using for coverage
testing.
This commit removes use_retry from the server config and provides a
public API for the user to manually accept/refuse/retry incoming
connections before a handshake begins, and inspect properties such as
an incoming connection's remote address and whether that address is
validated when doing so.

In quinn-proto, Incoming is made public, as well as Endpoint's accept/
refuse/retry methods which operate on it. The
DatagramEvent::NewConnection event is modified to return an incoming
but not yet accepted connection.

In quinn, awaiting Endpoint::accept now yields a new
quinn::Incoming type, rather than quinn::Connecting. The new
quinn::Incoming type has all the methods its quinn_proto equivalent has,
as well as an accept method to (fallibly) transition it into a
Connecting, and also refuse, retry, and ignore methods.

Furthermore, quinn::Incoming implements IntoFuture with the output type
Result<Connection, ConnectionError>>, which is the same as the Future
output type of Connecting. This lets server code which was
straightforwardly awaiting the result of quinn::Endpoint::accept work
with little to no modification.

The test accept_after_close was removed because the functionality it
was testing for no longer exists.
This commit adds a new --block option to the server example to
illustate in a simplified way the general structure one would use to
implement IP address blocking with the new accept/reject/retry API.

For example:

    cargo run --example server ./ --listen 127.0.0.1:4433 --stateless-retry --block 127.0.0.1:8065
    cargo run --example client https://127.0.0.1:4433/Cargo.toml --host localhost --bind 127.0.0.1:8065

One thing to note is that that example places the reject condition
before the retry condition. This expends slightly less effort rejecting
connections, but does create a blocked IP address oracle for an attacker
who can do address spoofing.
This commit adds a new --connection-limit option to the server example
to illustrate how a user could implement a limit to the number of
connections open at a time with the new "incoming" API and
Endpoint::open_connections method rather than with the now-removed
concurrent_connections ServerConfig parameter.
@gretchenfrage
Copy link
Contributor Author

Made requested tweaks and rebased over main.

@gretchenfrage
Copy link
Contributor Author

Tests passed, so should be ready to merge I think!

@lijunwangs
Copy link
Contributor

Tests passed, so should be ready to merge I think!

@gretchenfrage , @djc there are some unresolved conversations -- I think we just need to resolve them. I do not see any real outstanding issues.

@djc
Copy link
Member

djc commented Apr 3, 2024

@lijunwangs there is no need to keep litigating this -- please assume I'm well aware of the state.

@djc djc merged commit ff487f0 into quinn-rs:main Apr 3, 2024
8 checks passed
@djc
Copy link
Member

djc commented Apr 3, 2024

@gretchenfrage many thanks for all your work on this, and sorry for my slow feedback cycle(s).

@lijunwangs
Copy link
Contributor

@djc -- sorry for pushing for this -- it is a critical improvement for our use case. Thank you all!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Automatic Retry heuristics
5 participants