feat!: URLSession-based HTTP Client #636

jbelkins · 2023-12-27T16:52:09Z

Issue #

Description of changes

Provides URLSessionHTTPClient, which is an alternative to the CRT HTTP client that uses the Foundation library's URLSession networking client as its basis.

The advantage of this client over the existing AWS-provided HTTP client is that URLSession is deeply integrated into Apple operating systems and operates correctly and efficiently across all device states, network conditions, and application lifecycle states. The disadvantage is that it doesn't offer the same degree of configurability as the AWS HTTP client allows, and may not perform as well under heavy load.

The FoundationStreamBridge used to move streaming data from the SDK into Foundation uses Swift-Objective-C interoperability features that aren't available outside Apple platforms. Therefore, the URLSessionHTTPClient is unavailable elsewhere, such as Linux and Windows.

This client should meet the requirements of TN3135: Low-level networking on watchOS for running on watchOS, since it only uses what that note describes as "high-level networking".

Specific additions and changes:

The HttpClientEngine protocol is renamed to HTTPClient and revamped to match standard architecture.
The HttpClientConfiguration has been improved with better inline docs and a connect timeout setting.
The Stream protocol has had its throwing close() method replaced with non-throwing close() and closeWithError(_:). Conforming stream types have been adapted.
A FoundationStreamBridge type is provided that allows smithy-swift ReadableStream to be used as the streaming body for a Foundation URLRequest, which uses an older, Foundation-specific streaming mechanism.
The URLSessionHTTPClient is added.
- It converts a smithy-swift native SdkHttpRequest to a Foundation URLRequest, then executes it using URLSession
- URLSessionHTTPClient acts as the URLSessionDataDelegate to manage the response as it is received
- The request body is streamed into URLSession using the FoundationStreamBridge, while the response body is received in segments in URLSessionDataDelegate callbacks and streamed to the caller using a BufferedStream.
- URLSessionClient is used by default on all Apple platforms except Mac. The existing CRTClientEngine is used otherwise.
- Bidirectional streaming has been verified to operate correctly using the Transcribe integration test.

I have operationally tested this with a physical iOS 17 device and with a watchOS 10 device.

Also:

CI for tvOS (earliest & latest supported OS versions) has been added.
"Latest" version on CI is now Xcode 15.1 & iOS / tvOS 17.2.
Problems were experienced with testing Foundation streams on the simulators due to limits on thread usage in simulators. To prevent the simulator from hanging on stream tests, FoundationStreamBridgeTests are run on the Mac platform only.
The Transfer-Encoding: chunked header has been changed to match standard capitalization of header values.

Scope

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…atus

dayaffe · 2024-01-04T17:02:49Z

At a high-level, I am wondering if we have a breakdown of supported / not supported features between URLSession-based HTTP Client vs CRT?

Sources/ClientRuntime/Config/DefaultSDKRuntimeConfiguration.swift

Sources/ClientRuntime/Networking/Http/Middlewares/ContentLengthMiddleware.swift

jbelkins · 2024-01-04T17:26:45Z

At a high-level, I am wondering if we have a breakdown of supported / not supported features between URLSession-based HTTP Client vs CRT?

@dayaffe In general, features that are provided anywhere other than in the CRT HTTP client are supported / available on the new client. That means anything applied via the middleware stack, and it also means CRT-based features that aren't part of the CRT client, such as endpoint resolution, event stream parsing, code signing, etc.

Sources/ClientRuntime/Networking/Http/HttpClientConfiguration.swift

...ientRuntime/Networking/Http/URLSession/URLSessionConfiguration+HTTPClientConfiguration.swift

Sources/ClientRuntime/Networking/Http/URLSession/URLSessionHTTPClient.swift

Sources/ClientRuntime/Networking/Streaming/FileStream.swift

Sources/ClientRuntime/Networking/Http/URLSession/URLSessionHTTPClient.swift

dayaffe

Approving given that whitespace errors are fixed & waqar's comments are addressed. I'd also like to see a detailed breakdown of CRT-implemented features that will not work on this client + maybe the features URLSession provides that arent yet part of CRT, added to our official documentation so that customers know what should/shouldn't work on each platform.

jbelkins · 2024-01-05T20:25:24Z

@dayaffe I don't know of any features that wouldn't work on this client right now.
The only one potentially on the horizon is aws-chunked-encoding, but we don't have that on either client yet.

But as noted, anything that is implemented as a middleware on our operation stack should be utilized by any HTTP client.

sichanyoo

A couple questions

sichanyoo · 2024-01-08T14:57:23Z

Sources/ClientRuntime/EventStream/DefaultMessageDecoderStream.swift

@@ -11,22 +11,22 @@ extension EventStream {
    public struct DefaultMessageDecoderStream<Event: MessageUnmarshallable>: MessageDecoderStream {
        public typealias Element = Event

-        let stream: Stream
+        let stream: ReadableStream


Q: Why is changing the type from Stream to ReadableStream necessary?

Not necessary, per se, but is a good practice to constrain a type no more than needed.

The Stream protocol is just the combined ReadableStream and WritableStream protocols. But this type requires no WritableStream functionality, so that protocol is removed.

sichanyoo · 2024-01-08T14:58:29Z

Sources/ClientRuntime/EventStream/DefaultMessageEncoderStream.swift

-            fatalError("readToEndAsync() is not supported by AsyncStream backed streams")
+            var data = Data()
+            while let moreData = try await readAsync(upToCount: Int.max) {
+                data.append(moreData)
+            }
+            return data


Q: How come this method is now supported?

I don't remember exactly why it wasn't implemented. But we have a use for it in this PR, and as you can see it's pretty straightforward.

sichanyoo · 2024-01-08T14:59:48Z

Sources/ClientRuntime/Networking/Http/HttpClientConfiguration.swift

+    public init(
+        connectTimeout: TimeInterval? = nil,
+        defaultHeaders: Headers = Headers(),
+        protocolType: ProtocolType? = nil
+    ) {


Q: Why is it that the default value of ProtocolType is changed form .https to nil?

I will change this param back to protocolType: ProtocolType = .https to avoid breaking the existing API.
I also added a line to actually use the connectTimeout param.

* chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * Delete missed merge conflict marker. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com>

* chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * Missed merge conflict marker - deleted. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com>

* chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * Add customizations to auth resolve process. Add internal modeled layer for services (S3, EventBridge) that use rules-based auth scheme resolver. Rules-based auth scheme resolver work wrap-up. Wrap-up presign / presign-url refactor. Wrap-up refactor for fitting in rules-based auth scheme resolver. Update test cases to include new middlewares. Move requestSignature getter / setter/ key from aws middleware context extension to here. Also, add saving requestSignature to SignerMiddleware for consumption by event stream signing. * Add signEvent API to Signer protocol, and rename sign to signRequest. * Add mock auth scheme resolver, mock auth schemes, mock identity, mock identity resolver, and mock signer to use for middleware unit tests. * Add unit tests for AuthSchemeMiddleware and SignerMiddleware. * Update MockSigner to conform to modified Signer API with signEvent. * Rename directory containing mocks for auth tests from AuthTest to AuthTestUtil. * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * Update to reflect midleware generics change. * Delete unnessary line from test case. * Add CloudFront KeyValueStore as one of the services that use rules based auth scheme resolver customization. * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * Fix auth scheme middleware to save the selected auth scheme to middleware context by modifying the original context. Fixes transcribe streaming integration test where streaming signing flow was only accessing the original context and not the newly built one with selected auth scheme that was being passed to next middleware in line. * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * Address Josh's PR comments. * Merge updated CRT version from main into feat/test-suite. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com>

* chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * feat!: Eliminate service client protocols (#655) * feat: add support for flexible checksums on Normal payloads (#647) * chore: Update aws-crt-swift to 0.26.0 (#661) --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com>

* chore: Add new identity protocols. (#594) * Add new identity protocols. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Add signer protocol. (#598) * Add signer protocol & refactor HttpContext --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: Auth scheme changes (#601) * Add signer protocol. * Add http context changes. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: middleware changes (#605) * Add middlewares - AuthSchemeMiddleware and SignerMiddleware * Provide hook in auth scheme for signing properties customization. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: codegen changes (#609) * Add codegen for service specific auth scheme resolver protocol, service specific default auth scheme resolver struct, and service specific auth scheme resolver parameters struct. * Make ASR throw if passed in ASR params doesn't have region field for SigV4 auth scheme & fix ktlint issues. * Clean up middlewares. * Remove auth scheme and signing middlewares from operation stack of protocol tests. * Update test cases to include new middlewares. * Codegen more descriptive comment for empty service specific auth scheme resolver protocol. * Add codegen test for auth scheme resolver generation. * Move region in middleware context from sdk side to smithy side. * Remove AWSClientRuntime dependency - signingProperties will be set in auth scheme customization hooks instead. * Move auth schemes from service specific config to general AWS config. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: update epic branch with new changes in main (#617) * chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * Update test cases added from main to reflect sra identity & auth fields in middleware context & sra identity & auth middlewares in operation stack. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: customizations (#618) * Add customizations to auth resolve process. * Add signEvent API to Signer protocol, and rename sign to signRequest. * Resolve PR comments. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: merge most recent main into I&A project branch (#638) * chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * Delete missed merge conflict marker. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: tie up some loose ends (#645) * Rename runtime type and file in Kotlin side to match corresponding type in Swift side, now both called SignerMiddleware. * Change DefaultIdentityResolverConfiguration's identity resolvers member field to be Attributes so it can store multiple types of identity resolvers and return one with matching identity type. Necessary for supporting multiple types of identities down the line. * Detect newly added SigV4a trait and handle accordingly. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Merge updates from main into project epic branch (#658) * chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * Missed merge conflict marker - deleted. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat: test-suite (#651) * chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * Add customizations to auth resolve process. Add internal modeled layer for services (S3, EventBridge) that use rules-based auth scheme resolver. Rules-based auth scheme resolver work wrap-up. Wrap-up presign / presign-url refactor. Wrap-up refactor for fitting in rules-based auth scheme resolver. Update test cases to include new middlewares. Move requestSignature getter / setter/ key from aws middleware context extension to here. Also, add saving requestSignature to SignerMiddleware for consumption by event stream signing. * Add signEvent API to Signer protocol, and rename sign to signRequest. * Add mock auth scheme resolver, mock auth schemes, mock identity, mock identity resolver, and mock signer to use for middleware unit tests. * Add unit tests for AuthSchemeMiddleware and SignerMiddleware. * Update MockSigner to conform to modified Signer API with signEvent. * Rename directory containing mocks for auth tests from AuthTest to AuthTestUtil. * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * Update to reflect midleware generics change. * Delete unnessary line from test case. * Add CloudFront KeyValueStore as one of the services that use rules based auth scheme resolver customization. * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * Fix auth scheme middleware to save the selected auth scheme to middleware context by modifying the original context. Fixes transcribe streaming integration test where streaming signing flow was only accessing the original context and not the newly built one with selected auth scheme that was being passed to next middleware in line. * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * Address Josh's PR comments. * Merge updated CRT version from main into feat/test-suite. --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Merge latest changes from main into SRA I&A (#662) * chore: Require Swift 5.7, fix deprecation warnings (#600) * feat: support initial-response in RPC based event streams (#597) * chore: Updates version to 0.32.0 * chore: Add newline to README.md (#602) * feat: add limited support in smithy-swift for visionOS (#606) * feat: add support for requiresLength trait and Transfer-Encoding: Chunked (#604) * chore: Update to aws-crt-swift 0.15.0 (#607) * fix: content-length middleware should not error on event streams (#608) * chore: Updates version to 0.33.0 * chore: Improved downstream task (#568) * chore: Convert idempotency token middleware from closure to reusable type (#610) * fix: Update aws-crt-swift dependency to 0.17.0 (#612) * chore: Updates version to 0.34.0 * fix: Endpoint url should be nil if host or scheme is missing (#614) * fix: Pool HTTP connections based on scheme, host, and port (#615) * add default log level to initialize method (#616) * feat: add utility method for converting SdkHttpRequest to URLRequest. (#613) * Add extension constructor to URLRequest to convert SDKHttpRequest * Add preprocessor conditional import functionality to SwiftWriter. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * chore: Updates version to 0.35.0 * fix: Add a header to operation doc comments (#621) * remove unnecessary TODOs (#622) * fix: Codegen issues re: recursion, Swift keywords in unions (#623) * feat!: Replace the XML encoder with a custom Smithy implementation (#619) * feat!: Use closures for processing HTTP response (#624) * feat: add custom trait PaginationTruncationMember (#625) * allow isTruncated to be optional bool (#626) * chore: Updates version to 0.36.0 * chore: Run tvOS old & new in CI (#628) * fix: Fix Package.swift warning on Mac (#629) * chore: refactor HttpBody and ByteStream to be a single class ByteStream (#627) * chore: remove sync read in unused data extension (#630) * update smithy to 1.42.0 (#631) * chore: Updates version to 0.37.0 * chore: Update to aws-crt-swift 0.20.0 (#633) * fix: add back from method with fileHandle (#635) * fix!: Add no-op behavior for initialize methods of logging system. (#637) * Add no-op behavior for initialize methods if it isn't the first time being called. * Make LockingSystem threadsafe. * Make initialize methods async. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * feat!: URLSession-based HTTP Client (#636) * bump up CRT version to 0.22.0 (#639) * chore: Update version to 0.38.0 (#641) * chore: Empty commit (#643) * feat: add wrapper for checksums + unit tests (#642) * feat: Use the Foundation HTTP client by default on Mac (#646) * chore: Updates version to 0.39.0 * chore: Change MyURLQueryItem to SDKURLQueryItem (#652) * feat!: Provide HTTP request components by closure instead of protocol (#654) * fix: Don't retry modeled errors by default (#653) * feat!: Eliminate service client protocols (#655) * feat: add support for flexible checksums on Normal payloads (#647) * chore: Update aws-crt-swift to 0.26.0 (#661) --------- Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com> Co-authored-by: Sichan Yoo <chanyoo@amazon.com> * Change package import statements for AuthSchemeResolverGeneratorTests.kt to match other codegen tests in the repo. * Just for readability & understandability, match order of statements that add middlewares to middleware execution generator with the acutal order in codegen output. * Add documentation for MiddlewareExecutionGenerator.ContextAttributeCodegenFlowType. * Change getSize() of Attributes struct to a computed property. * Factor out and delete IdentityKind enum & use scheme ID instead to determine which identity resolver to return for a given auth scheme in default identity resolver config. * Change auth scheme middleware tests to reflect removal of IdentityKind. * Revert back codegen middleware stack order and add clarifying comment. * Change getOrNull() to orElseNull(null) in auth scheme resolver generator. * Add authSchemes & authSchemeResolver config properties to swift & kotlins sides. Refactor client runtime types for SRA auth for better organization. * Remove unused generic from AuthSchemeMiddleware and SignerMiddleware, then update codegen and codgen tests accordingly. * Fix errors found by generating SDK code. * Fix ktlint. * Fix error caught from CI. Updating SignerMiddleware's name property to SignerMiddleware from SigningMiddleware caused it to not be removed properly for generating protocol tests. * Move default auth scheme resolve logic from runtime to codegen following Steven's feedback on aws-sdk-swift PR. * Address Josh's minor comments. --------- Co-authored-by: Sichan Yoo <chanyoo@amazon.com> Co-authored-by: Josh Elkins <jbelkins@users.noreply.github.com> Co-authored-by: David Yaffe <dayaffe@amazon.com> Co-authored-by: AWS SDK Swift Automation <github-aws-sdk-swift-automation@amazon.com> Co-authored-by: Cyprien Ricque <48893621+CyprienRicque@users.noreply.github.com>

jbelkins added 16 commits June 2, 2023 15:18

URLSession client that works for non-streaming requests

09f8f63

Add a URLSession-based HTTP client

81faadd

Split readable & writable file streams

f1ffdd9

WIP

0031f1e

Merge branch 'main' into jbe/urlsession_client

d042993

Supports URLRequest body streaming

ce89d0b

Fix Transfer-Encoding header

93b31b4

Merge branch 'main' into jbe/urlsession_client

46426bd

Bidi streaming WIP

b4d2565

Merge branch 'main' into jbe/urlsession_client

5bb9740

Merge branch 'main' into jbe/urlsession_client

f209f1d

Works with transcribe streaming

28eb56c

Substantial cleanup

3ee7575

Add comments, logging

3870b30

Cleanup, docs, FoundationStreamBridge fixed

e15b52f

Fixes for Linux, Swift 5.7

9abdcbf

jbelkins changed the title ~~Jbe/urlsession client~~ feat: URLSession-based HTTP Client Dec 30, 2023

jbelkins marked this pull request as ready for review December 30, 2023 06:14

jbelkins added 12 commits December 30, 2023 00:15

Commit

c601988

Enable on Apple platforms only

42b2408

Write to output stream on thread

d478451

Close bridge on thread

2a5d86a

Timeout Swift unit tests at 15 min on CI

2191f63

Continue test if inputStream hasBytesAvailable

add91ef

Buffer ops on thread

5b2a6df

Open bridge in continuation

02652ce

Call all continuation blocks on GCD

bf38684

Use serial queue for bridge

bc19f4c

Cleanup

1b110d8

No waitUntilDone on open/close

45e6812

jbelkins requested a review from sichanyoo January 2, 2024 22:58

This was referenced Jan 2, 2024

chore: remove sync read in unused data extension #630

Merged

feat!: Use Foundation-based HTTP client on Apple platforms awslabs/aws-sdk-swift#1282

Merged

jbelkins added 2 commits January 2, 2024 19:47

Change HttpClientEngine protocol to match SRA

5e37faf

Configure correct HTTP client for platform

38eb272

jbelkins changed the title ~~feat: URLSession-based HTTP Client~~ feat!: URLSession-based HTTP Client Jan 3, 2024

jbelkins added 3 commits January 2, 2024 22:32

Fix tests

2cb5108

Exclude URLSessionConfiguration on Linux

2d79b03

Run stream bridge tests on macOS only, use actor to isolate stream st…

632395d

…atus

dayaffe reviewed Jan 4, 2024

View reviewed changes

Sources/ClientRuntime/Config/DefaultSDKRuntimeConfiguration.swift Show resolved Hide resolved

dayaffe reviewed Jan 4, 2024

View reviewed changes

Sources/ClientRuntime/Networking/Http/Middlewares/ContentLengthMiddleware.swift Show resolved Hide resolved

jbelkins mentioned this pull request Jan 4, 2024

chore: Use Xcode 15.1 as CI latest version awslabs/aws-sdk-swift#1286

Merged

waahm7 reviewed Jan 5, 2024

View reviewed changes

dayaffe approved these changes Jan 5, 2024

View reviewed changes

jbelkins added 2 commits January 5, 2024 13:56

Fix review comment issues

f8e97d3

Fix tests

b98fc6d

sichanyoo reviewed Jan 8, 2024

View reviewed changes

jbelkins added 2 commits January 9, 2024 15:15

Merge branch 'main' into jbe/urlsession_client

b7f3fab

Fix HttpClientConfiguration params

1214551

jbelkins merged commit 922d066 into main Jan 9, 2024
12 checks passed

jbelkins deleted the jbe/urlsession_client branch January 9, 2024 22:04

dayaffe pushed a commit that referenced this pull request Jan 11, 2024

feat!: URLSession-based HTTP Client (#636)

c275855

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat!: URLSession-based HTTP Client #636

feat!: URLSession-based HTTP Client #636

jbelkins commented Dec 27, 2023 •

edited

Loading

dayaffe commented Jan 4, 2024

jbelkins commented Jan 4, 2024

dayaffe left a comment

jbelkins commented Jan 5, 2024

sichanyoo left a comment

sichanyoo Jan 8, 2024

jbelkins Jan 9, 2024

sichanyoo Jan 8, 2024

jbelkins Jan 9, 2024

sichanyoo Jan 8, 2024

jbelkins Jan 9, 2024

feat!: URLSession-based HTTP Client #636

feat!: URLSession-based HTTP Client #636

Conversation

jbelkins commented Dec 27, 2023 • edited Loading

Issue #

Description of changes

Scope

dayaffe commented Jan 4, 2024

jbelkins commented Jan 4, 2024

dayaffe left a comment

Choose a reason for hiding this comment

jbelkins commented Jan 5, 2024

sichanyoo left a comment

Choose a reason for hiding this comment

sichanyoo Jan 8, 2024

Choose a reason for hiding this comment

jbelkins Jan 9, 2024

Choose a reason for hiding this comment

sichanyoo Jan 8, 2024

Choose a reason for hiding this comment

jbelkins Jan 9, 2024

Choose a reason for hiding this comment

sichanyoo Jan 8, 2024

Choose a reason for hiding this comment

jbelkins Jan 9, 2024

Choose a reason for hiding this comment

jbelkins commented Dec 27, 2023 •

edited

Loading