Problem running tests with Swift 5.5.2

[cbaker6]

I've been using this action with Swift 5.5.1 for the ParseSwift repo and have ran into very little issues. Every once in awhile 1/10 times it doesn't build the project and I need to rerun my CI; not a big deal.

For Swift 5.5.2, this issue is consistent as I can't run the tests at all. In addition, the Swift 5.5.2 takes a lot longer to build. I do have more async/await files in 5.5.2 (I ignore these in 5.5.1), but the difference in time is on 5.5.1 takes ~ 3 minutes to build and run the whole test suite and in 5.5.2 takes > 5 minutes to build, start the test and fail on the first test. My setup in actions is below:

windows:
    runs-on: windows-2019
    steps:
      - uses: actions/checkout@v2
      - uses: MaxDesiatov/swift-windows-action@v1
        with:
          swift-version: "5.5.2"
          shell-action: swift test --enable-test-discovery -v
      #    shell-action: swift test --enable-test-discovery --enable-code-coverage -v
      #- name: Upload coverage to Codecov
      #  uses: codecov/codecov-action@v2
      #  with:
      #    env_vars: WINDOWS
      #    fail_ci_if_error: false

Failed action in 5.5.2 with the above action setup: https://github.com/parse-community/Parse-Swift/runs/4843124003?check_suite_focus=true#step:3:7963

Successful action in 5.5.1: https://github.com/parse-community/Parse-Swift/runs/4842818376?check_suite_focus=true#step:3:7391

On both versions, the CI is build/test/passing in Linux using the swifty-linux-action which is why I believe it should build/pass in Windows. You can see the PR where I was testing this here: parse-community/Parse-Swift#320

[MaxDesiatov]

Thanks for reporting the issue! Would you mind trying if it also reproducible with this action? This would help us understand if the problem is specific to the action or to Swift 5.5.2 distribution for Windows in general https://github.com/compnerd/gha-setup-swift

[cbaker6]

Sure, I'll try and report back soon...

[cbaker6]

It looks like it's having the same issue: swift-5.5.2-RELEASE

Below is what I tried:

windows:
    runs-on: windows-2019
    steps:
      - uses: compnerd/gha-setup-swift@main
        with:
          branch: swift-5.5.2-release
          tag: 5.5.2-RELEASE

      - uses: actions/checkout@v2
      - run: swift build
      - run: swift test --enable-test-discovery --enable-code-coverage -v
    #steps:
    #  - uses: actions/checkout@v2
    #  - uses: MaxDesiatov/swift-windows-action@v1
    #    with:
    #      swift-version: "5.5.1"
    #      shell-action: swift test --enable-test-discovery --enable-code-coverage -v
      - name: Upload coverage to Codecov
        uses: codecov/codecov-action@v2
        with:
          env_vars: WINDOWS
          fail_ci_if_error: false

[MaxDesiatov]

I see, I guess the error message is the same as in this log? https://github.com/parse-community/Parse-Swift/runs/4843124003?check_suite_focus=true#step:3:7963

D:\a\Parse-Swift\Parse-Swift\.build\x86_64-unknown-windows-msvc\debug\ParseSwiftPackageTests.xctest
Test Suite 'All tests' started at 2022-01-17 15:56:14.625
Test Suite 'debug.xctest' started at 2022-01-17 15:56:14.628
Test Suite 'APICommandTests' started at 2022-01-17 15:56:14.628
Test Case 'APICommandTests.testApplicationIdHeader' started at 2022-01-17 15:56:14.628
Error: Process completed with exit code 1.

For Swift 5.5.2, this issue is consistent as I can't run the tests at all. In addition, the Swift 5.5.2 takes a lot longer to build. I do have more async/await files in 5.5.2 (I ignore these in 5.5.1), but the difference in time is on 5.5.1 takes ~ 3 minutes to build and run the whole test suite and in 5.5.2 takes > 5 minutes to build, start the test and fail on the first test.

I can hypothesise that the issue is with async/await, and this isn't a regression in non-async features in 5.5.2 but I'm not 100% sure. I wonder if disabling async/await on Windows completely would be a meaningful workaround. If disabling those async features helps you with building on Windows with 5.5.2, we at least would know that those features are the culprit.

@compnerd I know this may not be the right place to discuss this, but what would be the best way to gather information for a bug report on bugs.swift.org? This doesn't look like a bug related to GitHub Actions specifically, and seems to be a bug in the Windows toolchain. Are there any known issues with async/await on Windows that could be related? Thanks!

[cbaker6]

I see, I guess the error message is the same as in this log? https://github.com/parse-community/Parse-Swift/runs/4843124003?check_suite_focus=true#step:3:7963

Yes, this is the same log. If you want to see the PR I ran it on, it's here: parse-community/Parse-Swift#315

The PR is basically finished, I was just trying the Swift windows tests we are discussing.

I can hypothesise that the issue is with async/await, and this isn't a regression in non-async features in 5.5.2 but I'm not 100% sure. I wonder if disabling async/await on Windows completely would be a meaningful workaround. If disabling those async features helps you with building on Windows with 5.5.2, we at least would know that those features are the culprit.

What seems weird is it doesn't have any trouble building any of the async/await files or the test files that test them. Also, there's no async/await tests in APICommandTests, so I would expect that all of the tests in this file pass, but fail on something like, ParseAnanlyticsAsyncTests. I made sure to name files I conduct async and combine tests with Async and Combine respectively.

Update: It seems like the problem may not be related to async/await. I use #if compiler(>=5.5.2) && canImport(_Concurrency) to enable async/await across the SDK, so I switched all of this to #if compiler(>=5.6.2) && canImport(_Concurrency) and still got the same error. It built and tested fine for Linux and all of the Apple OS's. I'm assuming it didn't build the async files also as my codecov dropped significantly.

[compnerd]

I think that this might be a crash that I've seen with XCTest. I'm not sure what's the cause of it yet. It really needs a build and someone to investigate it. There's a bunch of overlap in terms of deadlines (5.5, 5.6, rebranch) and I need help to cover everything.