Skip to content

Commit

Permalink
chore(proxy): vendor a subset of rust-postgres (#9930)
Browse files Browse the repository at this point in the history
Our rust-postgres fork is getting messy. Mostly because proxy wants more
control over the raw protocol than tokio-postgres provides. As such,
it's diverging more and more. Storage and compute also make use of
rust-postgres, but in more normal usage, thus they don't need our crazy
changes.

Idea: 
* proxy maintains their subset
* other teams use a minimal patch set against upstream rust-postgres

Reviewing this code will be difficult. To implement it, I
1. Copied tokio-postgres, postgres-protocol and postgres-types from
https://github.com/neondatabase/rust-postgres/tree/00940fcdb57a8e99e805297b75839e7c4c7b1796
2. Updated their package names with the `2` suffix to make them compile
in the workspace.
3. Updated proxy to use those packages
4. Copied in the code from tokio-postgres-rustls 0.13 (with some patches
applied jbg/tokio-postgres-rustls#32
jbg/tokio-postgres-rustls#33)
5. Removed as much dead code as I could find in the vendored libraries
6. Updated the tokio-postgres-rustls code to use our existing channel
binding implementation
  • Loading branch information
conradludgate authored Nov 29, 2024
1 parent 3ffe6de commit 1d642d6
Show file tree
Hide file tree
Showing 58 changed files with 11,199 additions and 26 deletions.
3 changes: 3 additions & 0 deletions .config/hakari.toml
Original file line number Diff line number Diff line change
Expand Up @@ -46,6 +46,9 @@ workspace-members = [
"utils",
"wal_craft",
"walproposer",
"postgres-protocol2",
"postgres-types2",
"tokio-postgres2",
]

# Write out exact versions rather than a semver range. (Defaults to false.)
Expand Down
56 changes: 49 additions & 7 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

3 changes: 3 additions & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -35,6 +35,9 @@ members = [
"libs/walproposer",
"libs/wal_decoder",
"libs/postgres_initdb",
"libs/proxy/postgres-protocol2",
"libs/proxy/postgres-types2",
"libs/proxy/tokio-postgres2",
]

[workspace.package]
Expand Down
6 changes: 6 additions & 0 deletions libs/proxy/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
This directory contains libraries that are specific for proxy.

Currently, it contains a signficant fork/refactoring of rust-postgres that no longer reflects the API
of the original library. Since it was so significant, it made sense to upgrade it to it's own set of libraries.

Proxy needs unique access to the protocol, which explains why such heavy modifications were necessary.
21 changes: 21 additions & 0 deletions libs/proxy/postgres-protocol2/Cargo.toml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
[package]
name = "postgres-protocol2"
version = "0.1.0"
edition = "2018"
license = "MIT/Apache-2.0"

[dependencies]
base64 = "0.20"
byteorder.workspace = true
bytes.workspace = true
fallible-iterator.workspace = true
hmac.workspace = true
md-5 = "0.10"
memchr = "2.0"
rand.workspace = true
sha2.workspace = true
stringprep = "0.1"
tokio = { workspace = true, features = ["rt"] }

[dev-dependencies]
tokio = { workspace = true, features = ["full"] }
37 changes: 37 additions & 0 deletions libs/proxy/postgres-protocol2/src/authentication/mod.rs
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
//! Authentication protocol support.
use md5::{Digest, Md5};

pub mod sasl;

/// Hashes authentication information in a way suitable for use in response
/// to an `AuthenticationMd5Password` message.
///
/// The resulting string should be sent back to the database in a
/// `PasswordMessage` message.
#[inline]
pub fn md5_hash(username: &[u8], password: &[u8], salt: [u8; 4]) -> String {
let mut md5 = Md5::new();
md5.update(password);
md5.update(username);
let output = md5.finalize_reset();
md5.update(format!("{:x}", output));
md5.update(salt);
format!("md5{:x}", md5.finalize())
}

#[cfg(test)]
mod test {
use super::*;

#[test]
fn md5() {
let username = b"md5_user";
let password = b"password";
let salt = [0x2a, 0x3d, 0x8f, 0xe0];

assert_eq!(
md5_hash(username, password, salt),
"md562af4dd09bbb41884907a838a3233294"
);
}
}
Loading

1 comment on commit 1d642d6

@github-actions
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

6438 tests run: 6117 passed, 1 failed, 320 skipped (full report)


Failures on Postgres 16

  • test_sharded_ingest[github-actions-selfhosted-vanilla-1]: release-x86-64
# Run all failed tests locally:
scripts/pytest -vv -n $(nproc) -k "test_sharded_ingest[release-pg16-github-actions-selfhosted-vanilla-1]"
Flaky tests (2)

Postgres 17

Postgres 15

Code coverage* (full report)

  • functions: 30.2% (8178 of 27038 functions)
  • lines: 47.6% (64714 of 135838 lines)

* collected from Rust tests only


The comment gets automatically updated with the latest test results
1d642d6 at 2024-11-29T13:02:46.523Z :recycle:

Please sign in to comment.