Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add support for flexible checksums on streaming payloads #659

Merged
merged 80 commits into from
Mar 21, 2024
Merged
Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
80 commits
Select commit Hold shift + click to select a range
9be20ef
add support for flexible checksums on Normal payloads
dayaffe Jan 18, 2024
bcac65a
add support for Streaming flexible checksums
dayaffe Feb 7, 2024
b2f19e8
remove unnecessary function
dayaffe Feb 7, 2024
e51eea5
swiftlint
dayaffe Feb 7, 2024
d125440
remove duplicate test file
dayaffe Feb 7, 2024
38012af
comment out tests that arent ready
dayaffe Feb 8, 2024
64b4e4f
add ability to pass in priority list + tests from SEP
dayaffe Feb 8, 2024
0a461ce
change filestream to bufferedstream for unit testing
dayaffe Feb 8, 2024
66de094
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 8, 2024
ee76d94
change how response is updated
dayaffe Feb 8, 2024
dbfd9de
add support for sha1/sha256 to flexible checksum streams, update to c…
dayaffe Feb 9, 2024
efdd75d
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 9, 2024
427e7e4
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 9, 2024
09189f7
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 12, 2024
c4bbcfc
crt 0.27 requires logger change
dayaffe Feb 12, 2024
3e1b31d
fix logger errors
dayaffe Feb 12, 2024
a4d26d7
remove logger variable
dayaffe Feb 12, 2024
6d42b4f
update to crt 0.28.0 to fix typo
dayaffe Feb 12, 2024
b63aa96
fix tests that use modified MockOutput
dayaffe Feb 12, 2024
7d5ba6d
fix all swiftlint line is too long
dayaffe Feb 12, 2024
a440282
fix swiftlint
dayaffe Feb 12, 2024
620b313
last swiftlint
dayaffe Feb 12, 2024
5585d0b
try to fix CI by utilizing hasSpaceAvailable
dayaffe Feb 15, 2024
4df1b4c
try adding debugging statements
dayaffe Feb 15, 2024
5325161
try waiting for open stream status
dayaffe Feb 16, 2024
403d7c9
print stream status
dayaffe Feb 16, 2024
2575164
debugging xcode 14 issue
dayaffe Feb 16, 2024
69bb353
change to use chunkStorage and queue up chunks
dayaffe Feb 22, 2024
0b574eb
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 22, 2024
300f280
isChunked can check for eligible stream
dayaffe Feb 28, 2024
4b7b242
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 28, 2024
bc348fe
move x-amz-trailer header from setAwsChunkedHeader to flexible checks…
dayaffe Feb 28, 2024
ab771ee
specify CRT signer
dayaffe Feb 28, 2024
024e9cb
move setting chunked encoding headers to content length middleware
dayaffe Feb 28, 2024
ca44107
merge/refactor for identity & auth
dayaffe Feb 29, 2024
3d6ca01
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 29, 2024
5204bf5
fix failing test
dayaffe Feb 29, 2024
aaaad0a
swiftlint
dayaffe Feb 29, 2024
2c22cad
rename HashFunction to ChecksumAlgorithm for SRA
dayaffe Feb 29, 2024
5f3ba33
rename classes, use self.body, guard-else, copyright
dayaffe Feb 29, 2024
fcf3f79
remove getBody
dayaffe Feb 29, 2024
89da964
Merge branch 'main' into day/aws-chunked-encoding
dayaffe Feb 29, 2024
4be43d7
fix guard
dayaffe Feb 29, 2024
32de912
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 1, 2024
a03b94e
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 9, 2024
d49aeb8
Add timer to stream bridge thread before running run loop
jbelkins Mar 9, 2024
265d087
fix: Attach timer to run loop before starting it
jbelkins Mar 9, 2024
f175ca3
Merge branch 'jbe/run_loop_early_exit_fix' into day/aws-chunked-encoding
jbelkins Mar 9, 2024
f946689
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 11, 2024
3cee248
Eliminate extra HTTP calls in flexible checksum middleware
jbelkins Mar 12, 2024
540ca46
Middleware no longer calls next() more than once
jbelkins Mar 12, 2024
e76c30e
Add checksum algorithm tests
jbelkins Mar 12, 2024
e380706
Add chunked algorithm tests for crc32, crc32c
jbelkins Mar 12, 2024
fdece22
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 13, 2024
01e2726
Renamed HashFunctionTests to match class under test ChecksumAlgorithm
jbelkins Mar 13, 2024
e7c1195
Enforce order of stream bridge open & chunk sending
jbelkins Mar 14, 2024
389bc2b
Add logging
jbelkins Mar 14, 2024
5e35008
Chunk stream refactoring
jbelkins Mar 14, 2024
2809a84
More logging
jbelkins Mar 14, 2024
8353aab
Add more logging
jbelkins Mar 14, 2024
2ed0fa4
Even more logging
jbelkins Mar 14, 2024
0104e7b
Log all stream events
jbelkins Mar 14, 2024
67f52d0
Move prints to logger
jbelkins Mar 14, 2024
97bcbdc
Write chunks to stream as soon as they are received
jbelkins Mar 14, 2024
1db7911
Remove chunking from FoundationStreamBridge
jbelkins Mar 15, 2024
9de866a
Stream bridge logging cleanup
jbelkins Mar 15, 2024
c183e3a
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 15, 2024
130f9e8
Fix tests
jbelkins Mar 15, 2024
0ad3c2a
Fix lint
jbelkins Mar 15, 2024
75f36b5
URLSessionHTTPClient logging improvements
jbelkins Mar 15, 2024
871d456
Merge branch 'main' into day/aws-chunked-encoding
jbelkins Mar 15, 2024
da906f1
Merge branch 'main' of github.com:smithy-lang/smithy-swift into day/a…
jbelkins Mar 18, 2024
5ac8b03
Merge branch 'day/aws-chunked-encoding' of github.com:smithy-lang/smi…
jbelkins Mar 18, 2024
1bd9cd3
Move chunking logic into new composite AWSChunkedStream
jbelkins Mar 19, 2024
443432e
Fix tests
jbelkins Mar 19, 2024
8d54360
Merge remote-tracking branch 'origin/main' into day/aws-chunked-encoding
jbelkins Mar 19, 2024
afc1f81
Cleanup on URLSession send
jbelkins Mar 19, 2024
8c194fc
Chunking paced to sends on URLSession client; CRT client reverted
jbelkins Mar 20, 2024
028a133
Refactoring
jbelkins Mar 20, 2024
23c5ff3
Fix Swiftlint issue
jbelkins Mar 21, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -34,7 +34,7 @@ let package = Package(
.library(name: "SmithyTestUtil", targets: ["SmithyTestUtil"]),
],
dependencies: [
.package(url: "https://github.com/awslabs/aws-crt-swift.git", exact: "0.26.0"),
.package(url: "https://github.com/awslabs/aws-crt-swift.git", exact: "0.28.0"),
.package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"),
],
targets: [
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
//
// Copyright Amazon.com Inc. or its affiliates.
// All Rights Reserved.
//
// SPDX-License-Identifier: Apache-2.0
//

import AwsCommonRuntimeKit
dayaffe marked this conversation as resolved.
Show resolved Hide resolved

class AwsChunkedBufferedStream {
dayaffe marked this conversation as resolved.
Show resolved Hide resolved
jbelkins marked this conversation as resolved.
Show resolved Hide resolved
private var stream: BufferedStream
private var signingConfig: SigningConfig
private var previousSignature: String
private var trailingHeaders: Headers
private var chunkedReader: AwsChunkedReader

init(stream: BufferedStream, signingConfig: SigningConfig, previousSignature: String, trailingHeaders: Headers, checksumAlgorithm: HashFunction? = nil) {

Check warning on line 17 in Sources/ClientRuntime/Networking/AwsChunked/AwsChunkedBufferedStream.swift

View workflow job for this annotation

GitHub Actions / swiftlint

Line should be 120 characters or less; currently it has 157 characters (line_length)
self.stream = stream
self.signingConfig = signingConfig
self.previousSignature = previousSignature
self.trailingHeaders = trailingHeaders

self.chunkedReader = AwsChunkedReader(
stream: self.stream,
signingConfig: self.signingConfig,
previousSignature: self.previousSignature,
trailingHeaders: self.trailingHeaders,
checksum: checksumAlgorithm
)
}
}

extension AwsChunkedBufferedStream: Stream {

var position: Data.Index {
self.stream.position
}

var length: Int? {
self.stream.length
}

var isEmpty: Bool {
self.stream.isEmpty
}

var isSeekable: Bool {
self.stream.isSeekable
}

func read(upToCount count: Int) throws -> Data? {
try self.stream.read(upToCount: count)
}

func readAsync(upToCount count: Int) async throws -> Data? {
try await self.stream.readAsync(upToCount: count)
}

func readToEnd() throws -> Data? {
try self.stream.readToEnd()
}

func readToEndAsync() async throws -> Data? {
try await self.stream.readToEndAsync()
}

func write(contentsOf data: Data) throws {
try self.stream.write(contentsOf: data)
}

func close() {
self.stream.close()
}

func closeWithError(_ error: Error) {
self.stream.closeWithError(error)
}

}

extension AwsChunkedBufferedStream: AwsChunkedStream {

func getChunkedReader() -> AwsChunkedReader {
return self.chunkedReader
}

var checksumAlgorithm: HashFunction? {
get {
return self.chunkedReader.getChecksumAlgorithm()
}
set {
self.chunkedReader.setChecksumAlgorithm(checksum: newValue)
}
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
//
// Copyright Amazon.com Inc. or its affiliates.
// All Rights Reserved.
//
// SPDX-License-Identifier: Apache-2.0
//

import AwsCommonRuntimeKit

class AwsChunkedFileStream {
private var stream: FileStream
private var signingConfig: SigningConfig
private var previousSignature: String
private var trailingHeaders: Headers
private var chunkedReader: AwsChunkedReader

init(stream: FileStream, signingConfig: SigningConfig, previousSignature: String, trailingHeaders: Headers, checksumAlgorithm: HashFunction? = nil) {

Check warning on line 17 in Sources/ClientRuntime/Networking/AwsChunked/AwsChunkedFileStream.swift

View workflow job for this annotation

GitHub Actions / swiftlint

Line should be 120 characters or less; currently it has 153 characters (line_length)
self.stream = stream
self.signingConfig = signingConfig
self.previousSignature = previousSignature
self.trailingHeaders = trailingHeaders

self.chunkedReader = AwsChunkedReader(
stream: self.stream,
signingConfig: self.signingConfig,
previousSignature: self.previousSignature,
trailingHeaders: self.trailingHeaders,
checksum: checksumAlgorithm
)
}
}

extension AwsChunkedFileStream: Stream {

var position: Data.Index {
self.stream.position
}

var length: Int? {
self.stream.length
}

var isEmpty: Bool {
self.stream.isEmpty
}

var isSeekable: Bool {
self.stream.isSeekable
}

func read(upToCount count: Int) throws -> Data? {
try self.stream.read(upToCount: count)
}

func readAsync(upToCount count: Int) async throws -> Data? {
try await self.stream.readAsync(upToCount: count)
}

func readToEnd() throws -> Data? {
try self.stream.readToEnd()
}

func readToEndAsync() async throws -> Data? {
try await self.stream.readToEndAsync()
}

func write(contentsOf data: Data) throws {
try self.stream.write(contentsOf: data)
}

func close() {
self.stream.close()
}

func closeWithError(_ error: Error) {
self.stream.closeWithError(error)
}

}

extension AwsChunkedFileStream: AwsChunkedStream {

func getChunkedReader() -> AwsChunkedReader {
return self.chunkedReader
}

var checksumAlgorithm: HashFunction? {
get {
return self.chunkedReader.getChecksumAlgorithm()
}
set {
self.chunkedReader.setChecksumAlgorithm(checksum: newValue)
}
}
}
Loading
Loading