Add optional Rustls support #1099
Conversation
This commit adds a non-default feature flag to use Rustls instead of OpenSSL.
My personal motivation is to use Lychee on OpenBSD -current, where the
`openssl` crate frequently fails to link against the unreleased system
LibreSSL. Using the `vendored-openssl` feature helps with compilation, but
segfaults at runtime.
The commit adds three feature flags to the library, binary, benchmark, and all
examples:
- The `native-tls` feature flag toggles the `openssl` crate.
- The `rustls-tls` feature flag toggles the `rustls` crate.
- The `email-check` feature flag toggles the `check-if-email-exists` crate,
which is the only existing functionality currently incompatible with Rustls.
By default, `native-tls` and `email-check` are enabled. Thus, Lychee (bin and
lib) can be used as before unless default features are disabled.
To use the Rustls feature, pass `--no-default-features --features rustls` to
cargo check/build/test/..., e.g.,
$ cargo clippy --workspace --all-targets --no-default-features \ --features
rustls-tls -- --deny warnings
Checking email addresses requires both, `native-tls` and `email-check`, to be
enabled. Otherwise, email addresses are excluded.
The `email-check` feature flag is technically not necessary. I preferred it
over `not(rustls-tls)` because it's clearer and it addresses the AGPL license
issue lycheeverse#594. As far as I understand, a Lychee binary compiled without the
`email-check` feature could be distributed with file-based copyleft for the
MPL-licensed dependencies only. But that's out of scope here.
The benchmark shows a performance regression varying between 2% and 4.4% when
using Rustls instead of OpenSSL on my machine.
PS: The `ring` crate needs to be patched on OpenBSD 7.3 and later until the new
xonly patches have been upstreamed, see the `rust-ring` port.
By default, reqwest uses the webpki-roots crate with Rustls, effectively bundling Mozilla's root certificates. This commit uses the rustls-native-certs crate instead to use locally installed root certificates, to minimize the difference between the native-tls and rustls-tls features.
|
Couldn't take a look at the code yet, but I like the idea and I'm all for supporting more platforms. Thanks for the efforts so far. |
| pad = "0.1.6" | ||
| regex = "1.8.4" | ||
| reqwest = { version = "0.11.18", features = ["gzip"] } | ||
| reqwest = { version = "0.11.18", default-features = false, features = ["gzip", "json"] } |
There was a problem hiding this comment.
Interesting. Was the json feature missing before? Was it somehow implicitly activated?
There was a problem hiding this comment.
Good question. I assumed "json" was a default feature of reqwest, but it is not. In fact, I cannot see how it worked before. There is no other reqwest feature that enables the "json" feature, and it is necessary for the lychee::archive::wayback::get_wayback_link function.
| pulldown-cmark = "0.9.3" | ||
| regex = "1.8.4" | ||
| # Use trust-dns to avoid lookup failures on high concurrency | ||
| # https://github.com/seanmonstar/reqwest/issues/296 | ||
| reqwest = { version = "0.11.18", features = ["gzip", "trust-dns"] } | ||
| reqwest = { version = "0.11.18", default-features = false, features = ["gzip", "trust-dns"] } |
There was a problem hiding this comment.
This could potentially save a bit of compilation time.
Yes, I think we should document the feature flags somewhere. AFAIK there is no standard way to do this with rustdoc yet, see here. I've pushed a commit that adds a section to the README and comments to the Cargo.toml. |
Hello. I would like to discuss adding optional Rustls support to Lychee. I've prepared a pull request to show that's it's mostly a question of passing feature flags down to existing dependencies. What do you think?
This commit adds a non-default feature flag to use Rustls instead of OpenSSL.
My personal motivation is to use Lychee on OpenBSD -current, where the
opensslcrate frequently fails to link against the unreleased system LibreSSL. Using thevendored-opensslfeature helps with compilation, but segfaults at runtime.The commit adds three feature flags to the library, binary, benchmark, and all examples:
native-tlsfeature flag toggles theopensslcrate.rustls-tlsfeature flag toggles therustlscrate.email-checkfeature flag toggles thecheck-if-email-existscrate, which is the only existing functionality currently incompatible with Rustls.By default,
native-tlsandemail-checkare enabled. Thus, Lychee (bin and lib) can be used as before unless default features are disabled.To use the Rustls feature, pass
--no-default-features --features rustlsto cargo check/build/test/..., e.g.,Checking email addresses requires both,
native-tlsandemail-check, to be enabled. Otherwise, email addresses are excluded.The
email-checkfeature flag is technically not necessary. I preferred it overnot(rustls-tls)because it's clearer and it addresses the AGPL license issue #594. As far as I understand, a Lychee binary compiled without theemail-checkfeature could be distributed with file-based copyleft for the MPL-licensed dependencies only. But that's out of scope here.The benchmark shows a performance regression varying between 2% and 4.4% when using Rustls instead of OpenSSL on my machine.
PS: The
ringcrate needs to be patched on OpenBSD 7.3 and later until the new xonly patches have been upstreamed, see therust-ringport.