Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Make opened files visible in readdir #189

Merged
merged 6 commits into from
Apr 6, 2023
Merged

Make opened files visible in readdir #189

merged 6 commits into from
Apr 6, 2023

Conversation

monthonk
Copy link
Contributor

@monthonk monthonk commented Mar 31, 2023 •
edited
Loading

Currently, new files that are opened for write will not be visible in the file system until they are closed and their contents are uploaded to S3, but we want them to be visible as soon as we open the files.

To do this, we will store inodes of the opened files in its parent state and remove them once we finish writing. Then, on readdir, we can append these inodes to the results we get from LIST API to make the opened files visible in the file system.

Another problem is results ordering. Previously, we're using a double-ended queue to store the sorted LIST results but it doesn't work well with newly created files since we don't know their exact positions in paginated results. In this commit, we will use two queues, one for local results, and another one for remote results. Each queue is sorted by file names, then we compare the first element in both lists to decide which to return next.

A part of #27.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and I agree to the terms of the Developer Certificate of Origin (DCO).

@monthonk monthonk had a problem deploying to PR integration tests March 31, 2023 16:18 — with GitHub Actions Failure
@monthonk monthonk had a problem deploying to PR integration tests March 31, 2023 16:18 — with GitHub Actions Failure
@monthonk monthonk had a problem deploying to PR integration tests March 31, 2023 16:18 — with GitHub Actions Failure
@monthonk monthonk force-pushed the new_files_visibility branch from e4b3865 to dfe3380 Compare March 31, 2023 16:29
@monthonk monthonk temporarily deployed to PR integration tests March 31, 2023 16:29 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests March 31, 2023 16:29 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests March 31, 2023 16:29 — with GitHub Actions Inactive
@monthonk monthonk force-pushed the new_files_visibility branch from dfe3380 to 27ee44e Compare April 2, 2023 01:00
@monthonk monthonk temporarily deployed to PR integration tests April 2, 2023 01:00 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 2, 2023 01:00 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 2, 2023 01:00 — with GitHub Actions Inactive
jamesbornholt
jamesbornholt reviewed Apr 3, 2023
View reviewed changes
Copy link
Member

@jamesbornholt jamesbornholt left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Cool! A few suggestions.

mountpoint-s3/src/fs.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/inode.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/inode.rs
children: _,
writing_children,
} => writing_children.iter().map(|ino| {
let inode = self.inner.get(*ino)?;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We'll have to think later about the right behavior here when files can be unlinked (== inodes can disappear).

mountpoint-s3/tests/fuse_tests/readdir_test.rs Outdated Show resolved Hide resolved
mountpoint-s3/src/fs.rs Outdated Show resolved Hide resolved
@monthonk monthonk temporarily deployed to PR integration tests April 3, 2023 14:32 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 3, 2023 14:32 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 3, 2023 14:32 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 15:22 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 15:22 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 15:22 — with GitHub Actions Inactive
@monthonk monthonk force-pushed the new_files_visibility branch from 16683ca to 2389273 Compare April 4, 2023 16:39
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 16:39 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 16:39 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 4, 2023 16:39 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 5, 2023 09:36 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 5, 2023 09:36 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 5, 2023 09:36 — with GitHub Actions Inactive
@monthonk monthonk requested a review from jamesbornholt April 5, 2023 10:20
jamesbornholt
jamesbornholt reviewed Apr 5, 2023
View reviewed changes
mountpoint-s3/src/inode.rs
@@ -254,6 +253,22 @@ impl Superblock {
Ok(LookedUp { inode, stat })
}

/// Create a new write handle to be used for state transition
pub async fn write<OC: ObjectClient>(
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is there ever a case where we'd call this without immediately calling start_writing afterwards? If not we should just fold that transition into this constructor.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, we probably don't have such case, already moved it.

mountpoint-s3/tests/fuse_tests/readdir_test.rs Outdated
let mut map = HashMap::new();
let mut expected_list = Vec::new();
for i in 0..readdir_size * 4 {
let random_str = Alphanumeric.sample_string(&mut rand::thread_rng(), 5);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should use a fixed seed for the RNG here (and everywhere we use rand::thread_rng()) so test failures are reproducible. Then we should probably run this whole test in a loop, at least in the mock case, so we can get coverage of different random seeds.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated these tests.

mountpoint-s3/src/inode.rs
(None, Some(_)) => remote_locked.pop_front(),
(Some(_), None) => local_locked.pop_front(),
(Some(local_lookup), Some(remote_lookup)) => {
let ordering = local_lookup.inode.name().cmp(remote_lookup.inode.name());
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: lift the comment up here about how we need to consider the case where both inodes have the same name, and how we resolve that tie

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done.

mountpoint-s3/src/inode.rs
Comment on lines 665 to 669
if *next_token == ReaddirStreamState::Finished {
trace!(self=?self as *const _, "readdir finished");
return Ok(None);
return Ok(self.compare_and_get_next());
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we write a unit test (probably in this file) specifically to cover the cases around this logic?

  1. Remote directory is totally empty
  2. All local keys come after all remote keys

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Added these tests.

@monthonk monthonk had a problem deploying to PR integration tests April 6, 2023 13:16 — with GitHub Actions Failure
@monthonk monthonk had a problem deploying to PR integration tests April 6, 2023 13:16 — with GitHub Actions Failure
@monthonk monthonk had a problem deploying to PR integration tests April 6, 2023 13:16 — with GitHub Actions Failure
monthonk added 6 commits April 6, 2023 13:44
@monthonk
Make opened files visible in readdir
eb67a78 
Currently, new files that are opened for write will not be visible in
the file system until they are closed and their contents are uploaded to
S3, but we want them to be visible as soon as we open the files.

To do this, we will store inodes of the opened files in its parent state
and remove them once we finish writing. Then, on readdir, we can append
these inodes to the results we get from LIST API to make the opened files
visible in the file system.

Another problem is results ordering. Previously, we're using double-ended
queue to store the sorted LIST results but it doesn't work well with newly
created files since we don't know their exact positions in paginated
results. That's why it's also replaced by priority queue in this commit.

Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk
Address PR comments
a2cb441 
Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk
Use two VecDeques instead of BinaryHeap for results ordering
f279155 
Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk
Wrap the write state transition methods in WriteHandle
f299038 
Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk
Use random file names for readdir tests
d173336 
Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk
Improve testing
c9c3ea0 
Signed-off-by: Monthon Klongklaew <monthonk@amazon.com>
@monthonk monthonk force-pushed the new_files_visibility branch from a817ee0 to c9c3ea0 Compare April 6, 2023 13:51
@monthonk monthonk temporarily deployed to PR integration tests April 6, 2023 13:51 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 6, 2023 13:51 — with GitHub Actions Inactive
@monthonk monthonk temporarily deployed to PR integration tests April 6, 2023 13:51 — with GitHub Actions Inactive
@awslabs awslabs deleted a comment from Jcc2488 Apr 6, 2023
@jamesbornholt jamesbornholt merged commit c51e305 into awslabs:main Apr 6, 2023
@jamesbornholt
Copy link
Member

Nice!

@monthonk monthonk deleted the new_files_visibility branch April 6, 2023 15:09
passaro pushed a commit to passaro/mountpoint-s3 that referenced this pull request Jul 5, 2024
Update CRT submodules to latest releases
bdd38c7 
Submodule mountpoint-s3-crt-sys/crt/aws-c-cal 96c47e3..11fc684:
  > Make AES GCM more consistent cross platform (awslabs#189)
  > Pin AWS-LC until it's fixed for manylinux1 (awslabs#188)
  > Implement runtime check on libcrypto linkage (awslabs#186)
  > clang-format 18 (awslabs#187)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 06cf4d8..6d974f9:
  > cbor support  (awslabs#1131)
  > Fix default thread options for windows to not pin to any cpu_id (awslabs#1126)
  > Use CBMC 6.0.0 (awslabs#1128)
  > latest_submodules.py uses AWS-LC-FIPS releases in aws-crt-java (awslabs#1125)
  > Use CBMC version 5.95.1 (awslabs#1124)
  > clang-format 18 (awslabs#1113)
  > disable optimization was not working (awslabs#1123)
  > Fix memtracer bad assumptions on the size of stack trace (awslabs#1122)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 6588f9a..cb431ba:
  > test_helper.py improvements (awslabs#442)
  > Fix shutdown_callback or returning NULL contract for meta_request (awslabs#440)
  > BREAKING CHANGE: operation_name must be set for DEFAULT meta-requests (awslabs#439)
  > clang-format 18 (awslabs#438)
  > Auto - Update S3 Ruleset & Partition (awslabs#436)
Submodule mountpoint-s3-crt-sys/crt/aws-lc 92bf532..4368aaa:
  > Fix for loading JCA stripped private keys (#1658)
  > Prepare for release v1.30.1 (#1657)
  > Revert  `_CET_ENDBR` (#1656)
  > Close FD in Snapsafe test function (#1649)
  > Prepare for release v1.30.0 (#1646)
  > Snapsafe-type uniqueness breaking event detection (#1640)
  > Add EVP_md_null and SSL_set_ciphersuites (#1637)
  > Add de-randomized ML-KEM modes to experimental EVP API (#1578)
  > Patch for OpenVPN certificate setting behavioral difference (#1643)
  > Require newer assembler for _CET_ENDBR (#1641)
  > OpenVPN error codes, SSL_get_peer_signature_* funcs, and first patch file (#1584)
  > NIST.SP.800-56Cr2 One-Step Key Derivation (#1607)
  > Upstream merge 2024-06-13 (#1636)
  > More minor symbols for Ruby support (#1581)
  > Add support for NETSCAPE_SPKI_print (#1624)
  > align gcc version with curl's CI (#1633)
  > Fix spelling nits
  > Generated ASM files
  > Add Intel Indirect Branch Tracking support.
  > [EC] Unify point addition for P-256/384/521 (#1602)
  > Upstream merge 2024 06 03 (#1621)
  > Fix AES key size for AES256 in ABI test (#1629)
  > Move SSL_CIPHER_get_version test to SSLVersionTest.Version (#1631)
  > Use 'nasm' not 'yasm' (#1630)
  > Prepare for release 1.29.0 (#1626)
  > Implement SSL_CIPHER_get_version for recent TLS versions (#1627)
  > Add integration tests for OpenSSL-linking 3p modules (#1587)
  > Prevent non-constant-time code in Kyber-R3 and ML-KEM implementation (#1619)
  > Update ec2-test-framework to use gv2 (#1623)
  > Script for creating compilation database (#1617)
  > Fixes for building with `-pedantic` (#1608)
  > Fix SSL_BUILD_CHAIN_FLAG_IGNORE_ERROR behavior (#1620)
  > Update for FIPS documentation (#1610)
  > Disable CI for gcc-14/FIPS until relocation issue is resolved (#1622)
  > Add support for ocsp get id (#1609)
  > Add libevent to GitHub integration CI (#1615)
  > Upstream merge 2024 05 17 (#1600)
  > add back ASN1_dup with tests (#1591)
  > Remove special aarch64 valgrind logic (#1618)
  > Fix NTP integ test (#1616)
  > Pin aws-lc-rs integ to nightly-2024-05-22 (#1612)
  > Cleanse the right amount of bytes in HMAC. (#1613)
  > add support for X509_CRL_http_nbio (#1596)
  > Add `all_fuzz_tests` build target (#1605)
  > Fix mariadb ssl_crl patch (#1606)
Submodule mountpoint-s3-crt-sys/crt/s2n-tls 6d92b46..073c7b4:
  > bug: Fixing bash error (#4624)
  > chore: make cbmc proof build more strict by adding -Werror flag (#4606)
  > Perform 2-RTT Handshake to upgrade to PQ when possible (#4526)
  > test(bindings/s2n-tls): refactor testing::s2n-tls tests (#4613)
  > docs: add timeout note to blinding delay docs (#4621)
  > docs: Add back suggested FIPS + TLS1.3 policy (#4605)
  > ci: shallow clone musl repo (#4611)
  > example(bindings): add async ConfigResolver (#4477)
  > chore: use CBMC version 5.95.1 (#4586)
  > s2n-tls rust binding: expose selected application protocol (#4599)
  > test: add pcap testing crate (#4604)
  > testing(bindings): add new test helper (#4596)
  > chore(bindings): fix shebang in generate.sh (#4603)
  > fix(s2n_session_ticket_test): correct clock mocking (#4602)
  > Fix: update default cert chain for unit tests (#4582)
  > refactor(binding): more accurate naming for const str helper (#4601)
  > fix: error rather than empty cipher suites (#4597)
  > chore: update s2n_stuffer_printf CBMC harness (#4531)
  > ci(nix): Fix integ pq test in a devShell (#4576)
  > feature: new compatibility-focused security policy preferring ECDSA (#4579)
  > compliance: update generate_report.sh to point to compliance directory (#4588)
  > ci: fix cppcheck errors (#4589)
  > chore: cleanup duplicate duvet citations (#4587)
  > Merge pull request from GHSA-52xf-5p2m-9wrv
  > chore(bindings): release 0.2.7 (#4580)
  > fix: Validate received signature algorithm in EVP verify (#4574)
  > refactor: add try_compile feature probe for RSA-PSS signing (#4569)
  > feat: Configurable blinding (#4562)
  > docs: document s2n_cert_auth_type behavior (#4454)
  > fix: init implicit iv for serialization feature (#4572)
  > [Nix] adjust pytest retrys (#4558)
  > fix: cert verify test fix (#4545)
  > fix: update default security policies (#4523)
  > feat(bindings): Associate an application context with a Connection (#4563)
  > chore(bindings): version bump (#4566)
  > Additional test cases for s2n_constant_time_equals() (#4559)
  > test: backwards compatibility test for the serialization feature (#4548)
  > chore(bench): upgrade rustls (#4554)

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>
github-merge-queue bot pushed a commit that referenced this pull request Jul 10, 2024
@passaro
Update CRT libraries and set operation_name for DEFAULT meta-requests (…
a3c6f82 
…#935)

* Update CRT submodules to latest releases

Submodule mountpoint-s3-crt-sys/crt/aws-c-cal 96c47e3..11fc684:
  > Make AES GCM more consistent cross platform (#189)
  > Pin AWS-LC until it's fixed for manylinux1 (#188)
  > Implement runtime check on libcrypto linkage (#186)
  > clang-format 18 (#187)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 06cf4d8..6d974f9:
  > cbor support  (#1131)
  > Fix default thread options for windows to not pin to any cpu_id (#1126)
  > Use CBMC 6.0.0 (#1128)
  > latest_submodules.py uses AWS-LC-FIPS releases in aws-crt-java (#1125)
  > Use CBMC version 5.95.1 (#1124)
  > clang-format 18 (#1113)
  > disable optimization was not working (#1123)
  > Fix memtracer bad assumptions on the size of stack trace (#1122)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 6588f9a..cb431ba:
  > test_helper.py improvements (#442)
  > Fix shutdown_callback or returning NULL contract for meta_request (#440)
  > BREAKING CHANGE: operation_name must be set for DEFAULT meta-requests (#439)
  > clang-format 18 (#438)
  > Auto - Update S3 Ruleset & Partition (#436)
Submodule mountpoint-s3-crt-sys/crt/aws-lc 92bf532..4368aaa:
  > Fix for loading JCA stripped private keys (#1658)
  > Prepare for release v1.30.1 (#1657)
  > Revert  `_CET_ENDBR` (#1656)
  > Close FD in Snapsafe test function (#1649)
  > Prepare for release v1.30.0 (#1646)
  > Snapsafe-type uniqueness breaking event detection (#1640)
  > Add EVP_md_null and SSL_set_ciphersuites (#1637)
  > Add de-randomized ML-KEM modes to experimental EVP API (#1578)
  > Patch for OpenVPN certificate setting behavioral difference (#1643)
  > Require newer assembler for _CET_ENDBR (#1641)
  > OpenVPN error codes, SSL_get_peer_signature_* funcs, and first patch file (#1584)
  > NIST.SP.800-56Cr2 One-Step Key Derivation (#1607)
  > Upstream merge 2024-06-13 (#1636)
  > More minor symbols for Ruby support (#1581)
  > Add support for NETSCAPE_SPKI_print (#1624)
  > align gcc version with curl's CI (#1633)
  > Fix spelling nits
  > Generated ASM files
  > Add Intel Indirect Branch Tracking support.
  > [EC] Unify point addition for P-256/384/521 (#1602)
  > Upstream merge 2024 06 03 (#1621)
  > Fix AES key size for AES256 in ABI test (#1629)
  > Move SSL_CIPHER_get_version test to SSLVersionTest.Version (#1631)
  > Use 'nasm' not 'yasm' (#1630)
  > Prepare for release 1.29.0 (#1626)
  > Implement SSL_CIPHER_get_version for recent TLS versions (#1627)
  > Add integration tests for OpenSSL-linking 3p modules (#1587)
  > Prevent non-constant-time code in Kyber-R3 and ML-KEM implementation (#1619)
  > Update ec2-test-framework to use gv2 (#1623)
  > Script for creating compilation database (#1617)
  > Fixes for building with `-pedantic` (#1608)
  > Fix SSL_BUILD_CHAIN_FLAG_IGNORE_ERROR behavior (#1620)
  > Update for FIPS documentation (#1610)
  > Disable CI for gcc-14/FIPS until relocation issue is resolved (#1622)
  > Add support for ocsp get id (#1609)
  > Add libevent to GitHub integration CI (#1615)
  > Upstream merge 2024 05 17 (#1600)
  > add back ASN1_dup with tests (#1591)
  > Remove special aarch64 valgrind logic (#1618)
  > Fix NTP integ test (#1616)
  > Pin aws-lc-rs integ to nightly-2024-05-22 (#1612)
  > Cleanse the right amount of bytes in HMAC. (#1613)
  > add support for X509_CRL_http_nbio (#1596)
  > Add `all_fuzz_tests` build target (#1605)
  > Fix mariadb ssl_crl patch (#1606)
Submodule mountpoint-s3-crt-sys/crt/s2n-tls 6d92b46..073c7b4:
  > bug: Fixing bash error (#4624)
  > chore: make cbmc proof build more strict by adding -Werror flag (#4606)
  > Perform 2-RTT Handshake to upgrade to PQ when possible (#4526)
  > test(bindings/s2n-tls): refactor testing::s2n-tls tests (#4613)
  > docs: add timeout note to blinding delay docs (#4621)
  > docs: Add back suggested FIPS + TLS1.3 policy (#4605)
  > ci: shallow clone musl repo (#4611)
  > example(bindings): add async ConfigResolver (#4477)
  > chore: use CBMC version 5.95.1 (#4586)
  > s2n-tls rust binding: expose selected application protocol (#4599)
  > test: add pcap testing crate (#4604)
  > testing(bindings): add new test helper (#4596)
  > chore(bindings): fix shebang in generate.sh (#4603)
  > fix(s2n_session_ticket_test): correct clock mocking (#4602)
  > Fix: update default cert chain for unit tests (#4582)
  > refactor(binding): more accurate naming for const str helper (#4601)
  > fix: error rather than empty cipher suites (#4597)
  > chore: update s2n_stuffer_printf CBMC harness (#4531)
  > ci(nix): Fix integ pq test in a devShell (#4576)
  > feature: new compatibility-focused security policy preferring ECDSA (#4579)
  > compliance: update generate_report.sh to point to compliance directory (#4588)
  > ci: fix cppcheck errors (#4589)
  > chore: cleanup duplicate duvet citations (#4587)
  > Merge pull request from GHSA-52xf-5p2m-9wrv
  > chore(bindings): release 0.2.7 (#4580)
  > fix: Validate received signature algorithm in EVP verify (#4574)
  > refactor: add try_compile feature probe for RSA-PSS signing (#4569)
  > feat: Configurable blinding (#4562)
  > docs: document s2n_cert_auth_type behavior (#4454)
  > fix: init implicit iv for serialization feature (#4572)
  > [Nix] adjust pytest retrys (#4558)
  > fix: cert verify test fix (#4545)
  > fix: update default security policies (#4523)
  > feat(bindings): Associate an application context with a Connection (#4563)
  > chore(bindings): version bump (#4566)
  > Additional test cases for s2n_constant_time_equals() (#4559)
  > test: backwards compatibility test for the serialization feature (#4548)
  > chore(bench): upgrade rustls (#4554)

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Try to reduce package size

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Set operation_name when using MetaRequestType::Default

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Introduce S3Operation type

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

---------

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>
Co-authored-by: Alessandro Passaro <alexpax@amazon.com>
mhnap pushed a commit to mhnap/mountpoint-s3 that referenced this pull request Aug 1, 2024
@passaro
Update CRT libraries and set operation_name for DEFAULT meta-requests (…
52fa9ad 
…awslabs#935)

* Update CRT submodules to latest releases

Submodule mountpoint-s3-crt-sys/crt/aws-c-cal 96c47e3..11fc684:
  > Make AES GCM more consistent cross platform (awslabs#189)
  > Pin AWS-LC until it's fixed for manylinux1 (awslabs#188)
  > Implement runtime check on libcrypto linkage (awslabs#186)
  > clang-format 18 (awslabs#187)
Submodule mountpoint-s3-crt-sys/crt/aws-c-common 06cf4d8..6d974f9:
  > cbor support  (awslabs#1131)
  > Fix default thread options for windows to not pin to any cpu_id (awslabs#1126)
  > Use CBMC 6.0.0 (awslabs#1128)
  > latest_submodules.py uses AWS-LC-FIPS releases in aws-crt-java (awslabs#1125)
  > Use CBMC version 5.95.1 (awslabs#1124)
  > clang-format 18 (awslabs#1113)
  > disable optimization was not working (awslabs#1123)
  > Fix memtracer bad assumptions on the size of stack trace (awslabs#1122)
Submodule mountpoint-s3-crt-sys/crt/aws-c-s3 6588f9a..cb431ba:
  > test_helper.py improvements (awslabs#442)
  > Fix shutdown_callback or returning NULL contract for meta_request (awslabs#440)
  > BREAKING CHANGE: operation_name must be set for DEFAULT meta-requests (awslabs#439)
  > clang-format 18 (awslabs#438)
  > Auto - Update S3 Ruleset & Partition (awslabs#436)
Submodule mountpoint-s3-crt-sys/crt/aws-lc 92bf532..4368aaa:
  > Fix for loading JCA stripped private keys (#1658)
  > Prepare for release v1.30.1 (#1657)
  > Revert  `_CET_ENDBR` (#1656)
  > Close FD in Snapsafe test function (#1649)
  > Prepare for release v1.30.0 (#1646)
  > Snapsafe-type uniqueness breaking event detection (#1640)
  > Add EVP_md_null and SSL_set_ciphersuites (#1637)
  > Add de-randomized ML-KEM modes to experimental EVP API (#1578)
  > Patch for OpenVPN certificate setting behavioral difference (#1643)
  > Require newer assembler for _CET_ENDBR (#1641)
  > OpenVPN error codes, SSL_get_peer_signature_* funcs, and first patch file (#1584)
  > NIST.SP.800-56Cr2 One-Step Key Derivation (#1607)
  > Upstream merge 2024-06-13 (#1636)
  > More minor symbols for Ruby support (#1581)
  > Add support for NETSCAPE_SPKI_print (#1624)
  > align gcc version with curl's CI (#1633)
  > Fix spelling nits
  > Generated ASM files
  > Add Intel Indirect Branch Tracking support.
  > [EC] Unify point addition for P-256/384/521 (#1602)
  > Upstream merge 2024 06 03 (#1621)
  > Fix AES key size for AES256 in ABI test (#1629)
  > Move SSL_CIPHER_get_version test to SSLVersionTest.Version (#1631)
  > Use 'nasm' not 'yasm' (#1630)
  > Prepare for release 1.29.0 (#1626)
  > Implement SSL_CIPHER_get_version for recent TLS versions (#1627)
  > Add integration tests for OpenSSL-linking 3p modules (#1587)
  > Prevent non-constant-time code in Kyber-R3 and ML-KEM implementation (#1619)
  > Update ec2-test-framework to use gv2 (#1623)
  > Script for creating compilation database (#1617)
  > Fixes for building with `-pedantic` (#1608)
  > Fix SSL_BUILD_CHAIN_FLAG_IGNORE_ERROR behavior (#1620)
  > Update for FIPS documentation (#1610)
  > Disable CI for gcc-14/FIPS until relocation issue is resolved (#1622)
  > Add support for ocsp get id (#1609)
  > Add libevent to GitHub integration CI (#1615)
  > Upstream merge 2024 05 17 (#1600)
  > add back ASN1_dup with tests (#1591)
  > Remove special aarch64 valgrind logic (#1618)
  > Fix NTP integ test (#1616)
  > Pin aws-lc-rs integ to nightly-2024-05-22 (#1612)
  > Cleanse the right amount of bytes in HMAC. (#1613)
  > add support for X509_CRL_http_nbio (#1596)
  > Add `all_fuzz_tests` build target (#1605)
  > Fix mariadb ssl_crl patch (#1606)
Submodule mountpoint-s3-crt-sys/crt/s2n-tls 6d92b46..073c7b4:
  > bug: Fixing bash error (#4624)
  > chore: make cbmc proof build more strict by adding -Werror flag (#4606)
  > Perform 2-RTT Handshake to upgrade to PQ when possible (#4526)
  > test(bindings/s2n-tls): refactor testing::s2n-tls tests (#4613)
  > docs: add timeout note to blinding delay docs (#4621)
  > docs: Add back suggested FIPS + TLS1.3 policy (#4605)
  > ci: shallow clone musl repo (#4611)
  > example(bindings): add async ConfigResolver (#4477)
  > chore: use CBMC version 5.95.1 (#4586)
  > s2n-tls rust binding: expose selected application protocol (#4599)
  > test: add pcap testing crate (#4604)
  > testing(bindings): add new test helper (#4596)
  > chore(bindings): fix shebang in generate.sh (#4603)
  > fix(s2n_session_ticket_test): correct clock mocking (#4602)
  > Fix: update default cert chain for unit tests (#4582)
  > refactor(binding): more accurate naming for const str helper (#4601)
  > fix: error rather than empty cipher suites (#4597)
  > chore: update s2n_stuffer_printf CBMC harness (#4531)
  > ci(nix): Fix integ pq test in a devShell (#4576)
  > feature: new compatibility-focused security policy preferring ECDSA (#4579)
  > compliance: update generate_report.sh to point to compliance directory (#4588)
  > ci: fix cppcheck errors (#4589)
  > chore: cleanup duplicate duvet citations (#4587)
  > Merge pull request from GHSA-52xf-5p2m-9wrv
  > chore(bindings): release 0.2.7 (#4580)
  > fix: Validate received signature algorithm in EVP verify (#4574)
  > refactor: add try_compile feature probe for RSA-PSS signing (#4569)
  > feat: Configurable blinding (#4562)
  > docs: document s2n_cert_auth_type behavior (#4454)
  > fix: init implicit iv for serialization feature (#4572)
  > [Nix] adjust pytest retrys (#4558)
  > fix: cert verify test fix (#4545)
  > fix: update default security policies (#4523)
  > feat(bindings): Associate an application context with a Connection (#4563)
  > chore(bindings): version bump (#4566)
  > Additional test cases for s2n_constant_time_equals() (#4559)
  > test: backwards compatibility test for the serialization feature (#4548)
  > chore(bench): upgrade rustls (#4554)

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Try to reduce package size

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Set operation_name when using MetaRequestType::Default

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

* Introduce S3Operation type

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>

---------

Signed-off-by: Alessandro Passaro <alexpax@amazon.com>
Co-authored-by: Alessandro Passaro <alexpax@amazon.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@jamesbornholt jamesbornholt jamesbornholt approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@monthonk @jamesbornholt