feat: Add Swift Package Registry tools and config #1597

jbelkins · 2024-06-28T22:35:29Z

Issue #

Description of changes

Adds CLI tools for publishing either one or all packages to a static Swift Package Registry
In each generated service client, adds registry config and an embedded resource file with the package version

Note: Package.swift files for AWS runtime modules and service clients are temporarily renamed to Package.swift.txt. This is because Xcode hides any directory inside a Swift package that has another Swift package embedded in it. When it is time to ship these modules as their own Swift packages, these manifests will be renamed back to Package.swift.

New/existing dependencies impact assessment, if applicable

No new dependencies were added to this change.

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

jbelkins · 2024-06-29T02:54:02Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/Error.swift

-    var errorDescription: String? { message }
-
-    init(_ message: String) {
+public struct Error: LocalizedError {


As you will see below, many types like this one have been moved into AWSCLIUtils and been marked public, so that they can be shared between CLI utilities.

jbelkins · 2024-06-29T02:55:09Z

AWSSDKSwiftCLI/Package.swift

    ],
    dependencies: [
        .package(url: "https://github.com/apple/swift-argument-parser", from: "1.2.0"),
        .package(url: "https://github.com/apple/swift-package-manager", from: "0.6.0"),
        .package(url: "https://github.com/apple/swift-algorithms", from: "1.0.0"),
        .package(url: "https://github.com/apple/swift-log.git", from: "1.0.0"),
+        .package(url: "https://github.com/awslabs/aws-sdk-swift", from: "0.46.0"),


Our own AWS SDK for Swift is added as a dependency so that it can be used to perform the S3 & Cloudfront operations needed for publishing to the Registry.

jbelkins · 2024-06-29T02:57:07Z

AWSSDKSwiftCLI/Package.swift

                .product(name: "Algorithms", package: "swift-algorithms"),
-                .product(name: "Logging", package: "swift-log"),
            ],
            resources: [
                .process("Resources/Package.Base.swift"),
                .process("Resources/DocIndex.Base.md")
            ]
        ),


Two new CLI commands are created, spr-publish (for publishing one package to the Registry) and spr-multi-publish (for publishing the entire SDK). I elected to make them their own executables instead of adding more subcommands to AWSSDKSwiftCLI target above.

jbelkins · 2024-06-29T02:57:43Z

AWSSDKSwiftCLI/Package.swift

+                "spr-publish",
+            ]
+        ),
+        .target(


The SPR target is used for sharing registry-related code between the CLI commands.

jbelkins · 2024-06-29T02:58:36Z

AWSSDKSwiftCLI/Package.resolved

@@ -0,0 +1,95 @@
+{
+  "pins" : [


This file is committed so that when we run the command line tools, we all get the same versions of dependencies until we choose to update them.

jbelkins · 2024-06-29T02:59:54Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/Process+Utils.swift

@@ -29,14 +29,14 @@ extension Process {

    /// Returns the executable and arguments combined as a string
    var commandString: String {
-        let items = [executableURL?.path] + (arguments ?? [])
+        let items = [urlPath(executableURL)] + (arguments ?? [])


As you will see below, Foundation URL path is different between Mac & Linux, hence we use this wrapper method to get it.

jbelkins · 2024-06-29T03:01:27Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/Process+Utils.swift

    try _run(process)
    process.waitUntilExit()

+    let data = try stdOut.fileHandleForReading.readToEnd() ?? Data()


The readToEnd() method is used instead of the readabilityHandler above because sometimes the data was being returned before all of it had been written. readToEnd() guarantees that the file handle closes & all the data is read before it returns.

jbelkins · 2024-06-29T03:02:24Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/String+Utils.swift

@@ -37,3 +37,9 @@ extension String {
    /// Returns the string that represents a newline
    static var newline: Self { "\n" }
 }
+
+public func printError(_ items: Any..., separator: String = " ", terminator: String = "\n") throws {


A simple free function that acts like a print() statement but writes to stdErr.

jbelkins · 2024-06-29T03:04:19Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/URL+Path.swift

+public func urlPath(_ url: URL?) -> String? {
+    guard let url else { return nil }
+    return urlPath(url)
+}


It seems that on our supported Mac versions of Swift, you get the path from a URL with the path(percentEncoded:) method, while on Linux Foundation they still only have the .path property.

These two helper methods use the right one for the platform in use.

jbelkins · 2024-06-29T03:06:02Z

...DKSwiftCLI/Sources/AWSSDKSwiftCLI/Commands/AWSSDKSwiftCLI/Subcommands/GenerateDocIndex.swift

@@ -7,6 +7,7 @@

 import ArgumentParser
 import Foundation
+import AWSCLIUtils


Many utility types & functions were split into a new AWSCLIUtils module so they could be shared between the old AWSSDKSwiftCLI tool and the new registry commands.

Hence, a lot of imports to use that module.

jbelkins · 2024-06-29T03:07:50Z

AWSSDKSwiftCLI/Sources/AWSSDKSwiftCLI/Resources/Package.Base.swift

@@ -83,56 +83,56 @@ let package = Package(
                .awsSDKHTTPAuth,
                .awsSDKIdentity
            ],
-            path: "./Sources/Core/AWSClientRuntime",
+            path: "Sources/Core/AWSClientRuntime/Sources",


Both runtime modules and service clients have been arranged to match the internal layout of the standard Swift package. Paths in this file have been adjusted to accommodate.

Also, service clients now have a Resources subfolder which is included in the target.

jbelkins · 2024-06-29T03:09:56Z

AWSSDKSwiftCLI/Sources/SPR/Invalidate.swift

+import AWSCLIUtils
+import AWSCloudFront
+
+extension SPRPublisher {


Provides a function which is used to invalidate caching for the Registry version lists after they are updated.

On a spr-multi-publish run, only one invalidation is performed at the end, containing all version lists. This is because Cloudfront will start to throttle validations if they are done one path at a time for each package.

jbelkins · 2024-06-29T03:11:02Z

AWSSDKSwiftCLI/Sources/SPR/Models/Describe/Describe.swift

+
+import Foundation
+
+public struct Describe: Codable {


The many Codable structures below are used to read & write the standard JSON formatted data for registry version lists and version metadata.

jbelkins · 2024-06-29T03:13:31Z

AWSSDKSwiftCLI/Sources/SPR/Process+SPR.swift

+import PackageDescription
+import AWSCLIUtils
+
+extension Process {


These extensions run other utilities to perform steps in publication to the registry:

describe the contents of a package (used to verify package identity),

perform archiving of a package version to a .zip file, and

get the SHA-256 checksum for a file

jbelkins · 2024-06-29T03:34:08Z

Sources/Core/AWSSDKHTTPAuth/Package.swift

+
+import PackageDescription
+
+let package = Package(


Made this manifest for publishing AWSSDKHTTPAuth as its own package.

jbelkins · 2024-06-29T03:34:31Z

Sources/Core/AWSSDKIdentity/Package.swift

+import PackageDescription
+
+let package = Package(
+    name: "AWSSDKIdentity",


Made this manifest for publishing AWSSDKIdentity as its own package.

jbelkins · 2024-06-29T03:35:17Z

Sources/Core/AWSClientRuntime/Package.swift

+            resources: [.process("Resources")]
+        ),
+    ]
+)


You'll see many files moved below, as the runtime is organized into standard Swift package format.

jbelkins · 2024-06-29T03:37:15Z

...s/Core/AWSSDKEventStreamsAuth/Tests/AWSSDKEventStreamsAuthTests/SigV4EventSigningTests.swift

+import AWSSDKIdentity
+import AWSSDKHTTPAuth
+
+class SigV4EventSigningTests: XCTestCase {


The tests for SigV4 signing of event stream events has been moved from AWSSDKHTTPAuth to AWSSDKEventStreamsAuth to break a circular dependency between those two modules.

jbelkins · 2024-06-29T03:38:55Z

codegen/sdk-codegen/build.gradle.kts

@@ -161,13 +165,16 @@ fun discoverServices(): List<AwsService> {
            val serviceApi = service.getTrait(software.amazon.smithy.aws.traits.ServiceTrait::class.java).orNull()
                ?: error { "Expected aws.api#service trait attached to model ${file.absolutePath}" }
            val (name, version, _) = file.name.split(".")
+            val serviceClientVersions = Node.parse(rootProject.file("ServiceClientVersions.json").readText(Charset.forName("UTF-8")))
+            val packageVersion = serviceClientVersions.expectObjectNode().getStringMember(name).orNull()?.value ?:
+                serviceClientVersions.expectObjectNode().getStringMember("sdk").get().value


Reads the package version out of the new ServiceClientVersions.json file, then sets it in the smithy-build json so it can be code-generated into the service client.

jbelkins · 2024-06-29T03:40:38Z

...swift-codegen/src/main/kotlin/software/amazon/smithy/aws/swift/codegen/AWSSwiftDependency.kt

@@ -11,34 +11,38 @@ class AWSSwiftDependency {
        val AWS_SDK_IDENTITY = SwiftDependency(
            "AWSSDKIdentity",
            "main",
-            "0.1.0",


All these dependencies below have been updated with their registry scope and package name so they can be rendered as dependencies in Package.swift correctly.

jbelkins · 2024-06-29T03:42:09Z

...tlin/software/amazon/smithy/aws/swift/codegen/swiftintegrations/PackageVersionIntegration.kt

+import software.amazon.smithy.swift.codegen.integration.ProtocolGenerator
+import software.amazon.smithy.swift.codegen.integration.SwiftIntegration
+
+class PackageVersionIntegration : SwiftIntegration {


Writes the package version into the service client package's Resources. This can be used later for data collection purposes and for reference during publication.

jbelkins · 2024-06-29T03:42:55Z

...tlin/software/amazon/smithy/aws/swift/codegen/swiftintegrations/RegistryConfigIntegration.kt

+import software.amazon.smithy.swift.codegen.integration.ProtocolGenerator
+import software.amazon.smithy.swift.codegen.integration.SwiftIntegration
+
+class RegistryConfigIntegration : SwiftIntegration {


Writes the registry configuration file into the service client package, so the Swift build tools know where the registry is located.

jbelkins · 2024-06-29T03:43:57Z

scripts/ci_steps/codegen_sdk.sh

@@ -12,6 +12,7 @@ rm -rf Tests/Services/*
 ./gradlew --stop

 # Regenerate the SDK Package.swift with all services
+unset AWS_SWIFT_SDK_USE_LOCAL_DEPS


AWS_SWIFT_SDK_USE_LOCAL_DEPS is unset here & below before running the CLI tools so that the SDK used by the build tools doesn't attempt to use local smithy-swift.

sichanyoo

Some questions.

sichanyoo · 2024-07-08T16:52:01Z

AWSSDKSwiftCLI/Sources/AWSCLIUtils/URL+Path.swift

+@_disfavoredOverload
+public func urlPath(_ url: URL?) -> String? {
+    guard let url else { return nil }
+    return urlPath(url)
+}


Q: What's the point of this method? Why not just do guard check in the method that has compiler directives for OS type?

These two methods have very similar signatures but are distinct. (they differ only in whether the param type and return type are optionals or not). Mostly I want to have a non-optional return value when that is needed so I don't have to unwrap at the caller.

Swift was having problems resolving the right overload for the point of use so I added the @_disfavoredOverload annotation to help it find the right one.

sichanyoo · 2024-07-08T17:44:01Z

ServiceClientVersions.json

+  "sdk": "0.15.0",
+  "s3": "0.15.0"


Q: Where did 0.15.0 come from?

That is a number I was using for testing. It bears no relation to real release numbers. Before we would ship with this mechanism, there will be a script (not yet provided) that manages these numbers.

sichanyoo · 2024-07-08T17:54:24Z

scripts/ci_steps/codegen_sdk.sh

@@ -12,6 +12,7 @@ rm -rf Tests/Services/*
 ./gradlew --stop

 # Regenerate the SDK Package.swift with all services
+unset AWS_SWIFT_SDK_USE_LOCAL_DEPS


Q: Why shouldn't CLI tools not use local smithy-swift? Won't the pakages already be imported within GH action runner and made available locally?

The CLI now uses the latest stable aws-sdk-swift as a dependency (to perform the S3 & CloudFront operations needed to publish.)

If you leave that env var set, it will cause this dependency to break, because it will try to use a local copy of smithy-swift that isn't there.

For context, I expect to be removing the AWS_SWIFT_SDK_USE_LOCAL_DEPS env var entirely before GA. We'll use a different mechanism for local dependencies during development.

sichanyoo · 2024-07-08T17:57:04Z

...swift-codegen/src/main/kotlin/software/amazon/smithy/aws/swift/codegen/AWSSwiftDependency.kt

@@ -11,34 +11,38 @@ class AWSSwiftDependency {
        val AWS_SDK_IDENTITY = SwiftDependency(


Q: I see that github links are removed for the modules; were these git links for these dependencies even used anywhere?

Some dependencies (the ones that remain on git) still have a URL. The ones that will be published on the registry do not.

jbelkins added 16 commits June 22, 2024 12:59

feat: SPR CLI tool added

18c72f9

Reorg project into Swift packages

a5bf6fb

Fix list issues

ab35029

Fix Package.swift for runtime modules

ff8ec99

Merge remote-tracking branch 'origin/main' into jbe/spr

f301c24

Fix tests in runtime modules

6e1197c

Separate common CLI modules & SPR commands out from CLI tools

abc6e16

Multi-publish implemented

71f8a11

Add per-service versioning

071e733

Fix main Package.swift so it builds

7333369

Correct AWSClientRuntime Resource location

70d5f30

Add resources to target in main Package.swift

91220cb

Fix lint, fix CI failure

ddc9cc4

Wrap URL.path() to be platform dependent

eb409a0

Fix one more path()

5266077

Fix paths in Package.swift

4db4cef

jbelkins commented Jun 29, 2024

View reviewed changes

jbelkins added 3 commits June 28, 2024 22:58

Code cleanup

29a476d

Update Package.swift to match base

9bd23a9

Rename runtime manifests

a559104

jbelkins marked this pull request as ready for review July 1, 2024 15:55

jbelkins added 6 commits July 3, 2024 14:00

Merge remote-tracking branch 'origin/main' into jbe/spr

6eedca9

Merge branch 'main' into jbe/spr

d0f7420

Merge branch 'jbe/spr' of github.com:awslabs/aws-sdk-swift into jbe/spr

3aad806

Merge remote-tracking branch 'origin/main' into jbe/spr

8c825cc

Merge branch 'main' into jbe/spr

8ab7558

Merge branch 'main' into jbe/spr

1367871

sichanyoo self-requested a review July 8, 2024 16:01

jbelkins added 2 commits July 8, 2024 12:34

Merge branch 'main' into jbe/spr

63eb711

Merge branch 'jbe/spr' of github.com:awslabs/aws-sdk-swift into jbe/spr

1bd0c35

sichanyoo reviewed Jul 8, 2024

View reviewed changes

jbelkins added 3 commits July 9, 2024 12:10

Merge branch 'main' into jbe/spr

7fc9342

Merge remote-tracking branch 'origin/main' into jbe/spr

7a873a3

Merge remote-tracking branch 'origin' into jbe/spr

9a6903b

sichanyoo self-requested a review July 9, 2024 21:18

sichanyoo approved these changes Jul 9, 2024

View reviewed changes

jbelkins merged commit 0c5a63f into main Jul 9, 2024
29 checks passed

jbelkins deleted the jbe/spr branch July 9, 2024 22:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Add Swift Package Registry tools and config #1597

feat: Add Swift Package Registry tools and config #1597

jbelkins commented Jun 28, 2024 •

edited

Loading

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

jbelkins Jun 29, 2024

sichanyoo left a comment

sichanyoo Jul 8, 2024

jbelkins Jul 8, 2024

sichanyoo Jul 8, 2024

jbelkins Jul 8, 2024

sichanyoo Jul 8, 2024

jbelkins Jul 8, 2024

sichanyoo Jul 8, 2024

jbelkins Jul 8, 2024

		@@ -11,34 +11,38 @@ class AWSSwiftDependency {
		val AWS_SDK_IDENTITY = SwiftDependency(

feat: Add Swift Package Registry tools and config #1597

feat: Add Swift Package Registry tools and config #1597

Conversation

jbelkins commented Jun 28, 2024 • edited Loading

Issue #

Description of changes

New/existing dependencies impact assessment, if applicable

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sichanyoo left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jbelkins commented Jun 28, 2024 •

edited

Loading