Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: Add Swift Package Registry tools and config #1597

Merged
merged 30 commits into from
Jul 9, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
30 commits
Select commit Hold shift + click to select a range
18c72f9
feat: SPR CLI tool added
jbelkins Jun 22, 2024
a5bf6fb
Reorg project into Swift packages
jbelkins Jun 26, 2024
ab35029
Fix list issues
jbelkins Jun 27, 2024
ff8ec99
Fix Package.swift for runtime modules
jbelkins Jun 27, 2024
f301c24
Merge remote-tracking branch 'origin/main' into jbe/spr
jbelkins Jun 27, 2024
6e1197c
Fix tests in runtime modules
jbelkins Jun 27, 2024
abc6e16
Separate common CLI modules & SPR commands out from CLI tools
jbelkins Jun 27, 2024
71f8a11
Multi-publish implemented
jbelkins Jun 28, 2024
071e733
Add per-service versioning
jbelkins Jun 28, 2024
7333369
Fix main Package.swift so it builds
jbelkins Jun 28, 2024
70d5f30
Correct AWSClientRuntime Resource location
jbelkins Jun 28, 2024
91220cb
Add resources to target in main Package.swift
jbelkins Jun 28, 2024
ddc9cc4
Fix lint, fix CI failure
jbelkins Jun 29, 2024
eb409a0
Wrap URL.path() to be platform dependent
jbelkins Jun 29, 2024
5266077
Fix one more path()
jbelkins Jun 29, 2024
4db4cef
Fix paths in Package.swift
jbelkins Jun 29, 2024
29a476d
Code cleanup
jbelkins Jun 29, 2024
9bd23a9
Update Package.swift to match base
jbelkins Jun 29, 2024
a559104
Rename runtime manifests
jbelkins Jun 29, 2024
6eedca9
Merge remote-tracking branch 'origin/main' into jbe/spr
jbelkins Jul 3, 2024
d0f7420
Merge branch 'main' into jbe/spr
jbelkins Jul 5, 2024
3aad806
Merge branch 'jbe/spr' of github.com:awslabs/aws-sdk-swift into jbe/spr
jbelkins Jul 5, 2024
8c825cc
Merge remote-tracking branch 'origin/main' into jbe/spr
jbelkins Jul 5, 2024
8ab7558
Merge branch 'main' into jbe/spr
jbelkins Jul 5, 2024
1367871
Merge branch 'main' into jbe/spr
jbelkins Jul 5, 2024
63eb711
Merge branch 'main' into jbe/spr
jbelkins Jul 8, 2024
1bd0c35
Merge branch 'jbe/spr' of github.com:awslabs/aws-sdk-swift into jbe/spr
jbelkins Jul 8, 2024
7fc9342
Merge branch 'main' into jbe/spr
jbelkins Jul 9, 2024
7a873a3
Merge remote-tracking branch 'origin/main' into jbe/spr
jbelkins Jul 9, 2024
9a6903b
Merge remote-tracking branch 'origin' into jbe/spr
jbelkins Jul 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -35,3 +35,7 @@ codegen/protocol-test-codegen/smithy-build.json

# VS Code config files
.vscode/

# Allow the AWSSDKSwiftCLI Package.resolved
# while excluding all other Package.resolved files
!/AWSSDKSwiftCLI/Package.resolved
7 changes: 6 additions & 1 deletion .swiftlint.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,10 @@
included:
- Sources/Core
- Sources/Core/AWSClientRuntime/Sources
- Sources/Core/AWSSDKChecksums/Sources
- Sources/Core/AWSSDKCommon/Sources
- Sources/Core/AWSSDKEventStreamsAuth/Sources
- Sources/Core/AWSSDKHTTPAuth/Sources
- Sources/Core/AWSSDKIdentity/Sources

analyzer_rules:
- unused_import
Expand Down
95 changes: 95 additions & 0 deletions AWSSDKSwiftCLI/Package.resolved
Original file line number Diff line number Diff line change
@@ -0,0 +1,95 @@
{
"pins" : [
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This file is committed so that when we run the command line tools, we all get the same versions of dependencies until we choose to update them.

{
"identity" : "aws-crt-swift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/awslabs/aws-crt-swift",
"state" : {
"revision" : "b6380f683b31072d77b601c88674461c11bcad5a",
"version" : "0.30.0"
}
},
{
"identity" : "aws-sdk-swift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/awslabs/aws-sdk-swift",
"state" : {
"revision" : "3c20e0be8c8246de8b8e04372404ef1f90be71b6",
"version" : "0.46.0"
}
},
{
"identity" : "smithy-swift",
"kind" : "remoteSourceControl",
"location" : "https://github.com/smithy-lang/smithy-swift",
"state" : {
"revision" : "b2322a067f85c230f17c80be8a67dd543454b081",
"version" : "0.51.0"
}
},
{
"identity" : "swift-algorithms",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-algorithms",
"state" : {
"revision" : "f6919dfc309e7f1b56224378b11e28bab5bccc42",
"version" : "1.2.0"
}
},
{
"identity" : "swift-argument-parser",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-argument-parser",
"state" : {
"revision" : "46989693916f56d1186bd59ac15124caef896560",
"version" : "1.3.1"
}
},
{
"identity" : "swift-llbuild",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-llbuild.git",
"state" : {
"revision" : "fb7ebf0b06c0d7c45ca8e18b3371424503a38b5c",
"version" : "0.3.0"
}
},
{
"identity" : "swift-log",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-log.git",
"state" : {
"revision" : "e97a6fcb1ab07462881ac165fdbb37f067e205d5",
"version" : "1.5.4"
}
},
{
"identity" : "swift-numerics",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-numerics.git",
"state" : {
"revision" : "0a5bc04095a675662cf24757cc0640aa2204253b",
"version" : "1.0.2"
}
},
{
"identity" : "swift-package-manager",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-package-manager",
"state" : {
"revision" : "f5ea3972d7d6c574e8bb16a19b2a7bca98ea131b",
"version" : "0.6.0"
}
},
{
"identity" : "swift-tools-support-core",
"kind" : "remoteSourceControl",
"location" : "https://github.com/apple/swift-tools-support-core.git",
"state" : {
"revision" : "98a5916a811fcaaed770f1ed812e9405be762945",
"version" : "0.1.0"
}
}
],
"version" : 2
}
39 changes: 36 additions & 3 deletions AWSSDKSwiftCLI/Package.swift
Original file line number Diff line number Diff line change
Expand Up @@ -5,28 +5,61 @@ import PackageDescription
let package = Package(
name: "AWSSDKSwiftCLI",
platforms: [
.macOS(.v10_15)
.macOS(.v13)
],
dependencies: [
.package(url: "https://github.com/apple/swift-argument-parser", from: "1.2.0"),
.package(url: "https://github.com/apple/swift-package-manager", from: "0.6.0"),
.package(url: "https://github.com/apple/swift-algorithms", from: "1.0.0"),
.package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"),
.package(url: "https://github.com/awslabs/aws-sdk-swift", from: "0.46.0"),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Our own AWS SDK for Swift is added as a dependency so that it can be used to perform the S3 & Cloudfront operations needed for publishing to the Registry.

],
targets: [
.executableTarget(
name: "AWSSDKSwiftCLI",
dependencies: [
"AWSCLIUtils",
.product(name: "ArgumentParser", package: "swift-argument-parser"),
.product(name: "PackageDescription", package: "swift-package-manager"),
.product(name: "Algorithms", package: "swift-algorithms"),
.product(name: "Logging", package: "swift-log"),
],
resources: [
.process("Resources/Package.Base.swift"),
.process("Resources/DocIndex.Base.md")
]
),
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two new CLI commands are created, spr-publish (for publishing one package to the Registry) and spr-multi-publish (for publishing the entire SDK). I elected to make them their own executables instead of adding more subcommands to AWSSDKSwiftCLI target above.

.executableTarget(
name: "spr-publish",
dependencies: [
"SPR",
"AWSCLIUtils",
.product(name: "ArgumentParser", package: "swift-argument-parser"),
]
),
.executableTarget(
name: "spr-multi-publish",
dependencies: [
"SPR",
"AWSCLIUtils",
"spr-publish",
]
),
.target(
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SPR target is used for sharing registry-related code between the CLI commands.

name: "SPR",
dependencies: [
"AWSCLIUtils",
.product(name: "ArgumentParser", package: "swift-argument-parser"),
.product(name: "AWSS3", package: "aws-sdk-swift"),
.product(name: "AWSCloudFront", package: "aws-sdk-swift"),
]
),
.target(
name: "AWSCLIUtils",
dependencies: [
.product(name: "ArgumentParser", package: "swift-argument-parser"),
.product(name: "Logging", package: "swift-log"),
.product(name: "PackageDescription", package: "swift-package-manager"),
]
),
.testTarget(
name: "AWSSDKSwiftCLITests",
dependencies: ["AWSSDKSwiftCLI"]
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -7,11 +7,11 @@

import Foundation

struct Error: LocalizedError {
var message: String
var errorDescription: String? { message }
init(_ message: String) {
public struct Error: LocalizedError {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you will see below, many types like this one have been moved into AWSCLIUtils and been marked public, so that they can be shared between CLI utilities.

public var message: String
public var errorDescription: String? { message }

public init(_ message: String) {
self.message = message
}
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

import Foundation

extension FileManager {
public extension FileManager {
/// Changes the working directory to the provided path
func changeWorkingDirectory(_ path: String) throws {
log("Changing working directory to: \(path)")
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,11 @@ import Logging
struct Logger {
static let standard = Logging.Logger(
label: "com.aws.sdk.swift.cli",
factory: { StreamLogHandler.standardOutput(label: $0) }
factory: {
var logger = StreamLogHandler.standardOutput(label: $0)
logger.logLevel = .info
return logger
}
)
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ import Foundation
import PackageDescription
import struct ArgumentParser.ExitCode

extension Process {
public extension Process {
/// A struct to create processes for executing git commands.
struct Git {
/// Returns a process for executing git commands.
Expand All @@ -18,31 +18,31 @@ extension Process {
}

/// Returns a process for executing `git status`
func status() -> Process {
public func status() -> Process {
gitProcess(["status"])
}

/// Returns a process for executing `git diff <a>..<b> --quiet`
/// This is used for determining if `<a>` is different from `<b>` and therefore
func diff(_ a: String, _ b: String) -> Process {
public func diff(_ a: String, _ b: String) -> Process {
gitProcess(["diff", "\(a)..\(b)", "--quiet"])
}

/// Returns a process for executing `git add <args>`
/// This is used for staging specific files.
func add(_ files: [String]) -> Process {
public func add(_ files: [String]) -> Process {
gitProcess(["add"] + files)
}

/// Returns a process for executing `git commit -m <message>`
/// This is used for committing changes with the provided message
func commit(_ message: String) -> Process {
public func commit(_ message: String) -> Process {
gitProcess(["commit", "-m", message])
}

/// Returns a process for executing `git tag -a <version> -m <message>`
/// This is used for creating a tag with the provided version and message.
func tag(_ version: Version, _ message: String) -> Process {
public func tag(_ version: Version, _ message: String) -> Process {
gitProcess(["tag", "-a", "\(version)", "-m", message])
}

Expand All @@ -69,7 +69,7 @@ extension Process {
static var git: Git { Git() }
}

extension Process.Git {
public extension Process.Git {
/// Returns true if the provided commits/branches/trees are different, otherwise returns false
func diffHasChanges(_ a: String, _ b: String) throws -> Bool {
let task = diff(a, b)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,15 +8,15 @@
import Foundation
import PackageDescription

extension Process {
public extension Process {
struct Swift {
/// Returns a process for executing swift commands.
private func swiftProcess(_ args: [String]) -> Process {
Process(["swift"] + args)
}

/// Returns a process for executing `swift test`
func test() -> Process {
public func test() -> Process {
swiftProcess(["test"])
}
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@
import Foundation
import ArgumentParser

extension Process {
public extension Process {
/// Creates a process using `/usr/bin/env` as the executable
/// This makes it easy to create a process for any command, as long as the corresponding executable exists in the PATH.
///
Expand All @@ -29,14 +29,14 @@ extension Process {

/// Returns the executable and arguments combined as a string
var commandString: String {
let items = [executableURL?.path] + (arguments ?? [])
let items = [urlPath(executableURL)] + (arguments ?? [])
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As you will see below, Foundation URL path is different between Mac & Linux, hence we use this wrapper method to get it.

return items
.compactMap { $0 }
.joined(separator: " ")
}
}

func _run(_ process: Process) throws {
public func _run(_ process: Process) throws {
// If debug and we have a non-nil test runner, then use that
// This allows developers to intercept processes when they run to assert that it is the expected process
#if DEBUG
Expand All @@ -48,30 +48,31 @@ func _run(_ process: Process) throws {
try ProcessRunner.standard.run(process)
}

func _runReturningStdOut(_ process: Process) throws -> String? {
public func _runReturningStdOut(_ process: Process) throws -> String? {
let stdOut = Pipe()
process.standardOutput = stdOut

var data = Data()
stdOut.fileHandleForReading.readabilityHandler = { handle in
data += handle.availableData
}

try _run(process)
process.waitUntilExit()

let data = try stdOut.fileHandleForReading.readToEnd() ?? Data()
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The readToEnd() method is used instead of the readabilityHandler above because sometimes the data was being returned before all of it had been written. readToEnd() guarantees that the file handle closes & all the data is read before it returns.

return String(data: data, encoding: .utf8)
}

/// A simple struct that runs a process
struct ProcessRunner {
let run: (Process) throws -> Void
public struct ProcessRunner {

public init(_ run: @escaping (Process) throws -> Void) {
self.run = run
}

public let run: (Process) throws -> Void

/// Creates the standard runner to be used by the release version of this CLI
///
/// Runs the process and prints out the process's full command.
static let standard = ProcessRunner { process in
log("Running process: \(process.commandString)")
public static let standard = ProcessRunner { process in
log(level: .debug, "Running process: \(process.commandString)")
try process.run()
process.waitUntilExit()
let exitCode = ExitCode(process.terminationStatus)
Expand All @@ -82,6 +83,6 @@ struct ProcessRunner {

#if DEBUG
// Set this to a non-nil value in tests to intercept when a process is run
static var testRunner: ProcessRunner? = nil
public static var testRunner: ProcessRunner? = nil
#endif
}
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@

import Foundation

extension String {
public extension String {
/// Returns the receiver wrapped in the provided string
///
/// ```swift
Expand Down Expand Up @@ -37,3 +37,9 @@ extension String {
/// Returns the string that represents a newline
static var newline: Self { "\n" }
}

public func printError(_ items: Any..., separator: String = " ", terminator: String = "\n") throws {
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A simple free function that acts like a print() statement but writes to stdErr.

var s = ""
print(items, separator: separator, terminator: terminator, to: &s)
try FileHandle.standardError.write(contentsOf: Data(s.utf8))
}
Loading
Loading