Calling waitUntil in a QuickConfiguration times out for AsyncSpec

[CraigSiemens]

• I have read CONTRIBUTING and have done my best to follow them.

What did you do?

If a QuickConfiguration contains a call to waitUntil, it will always timeout when running an AsyncSpec. In our case, we're using beforeSuite and afterSuite to call asynchronous methods to save and restore the values in the keychain to avoid

Below is the simplest way I could find to reproduce it. The test will always fail with the message

class AsyncConfiguration: QuickConfiguration {
    override class func configure(_ configuration: QCKConfiguration) {
        configuration.beforeSuite {
            waitUntil { done in
                done()
            }
        }
    }
}

class AsyncConfigurationSpec: AsyncSpec {
    override class func spec() {
        it("should be called") {}
    }
}

What did you expect to happen?

waitUntil doesn't timeout when the test is a subclass of QuickSpec.

What actually happened instead?

The closure passed to waitUntil is never called. Neither a breakpoint or a print are called until after waitUntil has timed out.

Increasing the timeout interval has no affect. The test will always wait the full timeout before failing.

The test failed with the following message.

should be called(): Waited more than 1.0 second

Environment

List the software versions you're using:

• Quick: 7.4.0
• Nimble: 13.2.0
• Xcode Version: 15.2
• Swift Version: 5.9

Please also mention which package manager you used and its version. Delete the
other package managers in this list:

• Swift Package Manager - Swift 5.9.0

Project that demonstrates the issue

I reproduced it with the Quick repo, since I wasn't sure which repo was the cause of the issue.

1. Clone Quick
2. Copy the following diff
3. pbpaste | git apply
4. Open Quick and run the tests.

diff --git a/Package.swift b/Package.swift
index 264a133..a268321 100644
--- a/Package.swift
+++ b/Package.swift
@@ -35,6 +35,10 @@ let package = Package(
                 name: "QuickIssue853RegressionTests",
                 dependencies: [ "Quick" ]
             ),
+            .testTarget(
+                name: "QuickAsyncConfigurationTests",
+                dependencies: [ "Quick", "Nimble" ]
+            ),
         ]
 #if os(macOS)
         targets.append(contentsOf: [
diff --git a/Package@swift-5.9.swift b/Package@swift-5.9.swift
index 4ede907..5daa938 100644
--- a/Package@swift-5.9.swift
+++ b/Package@swift-5.9.swift
@@ -35,6 +35,10 @@ let package = Package(
                 name: "QuickIssue853RegressionTests",
                 dependencies: [ "Quick" ]
             ),
+            .testTarget(
+                name: "QuickAsyncConfigurationTests",
+                dependencies: [ "Quick", "Nimble" ]
+            ),
         ]
 #if os(macOS)
         targets.append(contentsOf: [
diff --git a/Tests/QuickAsyncConfigurationTests/AsyncConfigurationSpec.swift b/Tests/QuickAsyncConfigurationTests/AsyncConfigurationSpec.swift
index ea0cc4b..2cfa8a0 100644
--- a/Tests/QuickAsyncConfigurationTests/AsyncConfigurationSpec.swift
+++ b/Tests/QuickAsyncConfigurationTests/AsyncConfigurationSpec.swift
@@ -2,30 +2,18 @@ import XCTest
 import Quick
 import Nimble
 
-class AsyncConfigurationSpec: AsyncSpec {
-    override class func spec() {
-        beforeEach {
-            FunctionalTests_Configuration_AfterEachWasExecuted = false
-        }
-        it("is executed before the configuration afterEach") {
-            expect(FunctionalTests_Configuration_AfterEachWasExecuted).to(beFalsy())
+class AsyncConfiguration: QuickConfiguration {
+    override class func configure(_ configuration: QCKConfiguration) {
+        configuration.beforeEach {
+            waitUntil { done in
+                done()
+            }
         }
     }
 }
 
-final class Configuration_AfterEachAsyncTests: XCTestCase, XCTestCaseProvider {
-    static var allTests: [(String, (Configuration_AfterEachAsyncTests) -> () throws -> Void)] {
-        return [
-            ("testExampleIsRunAfterTheConfigurationBeforeEachIsExecuted", testExampleIsRunAfterTheConfigurationBeforeEachIsExecuted),
-        ]
-    }
-
-    func testExampleIsRunAfterTheConfigurationBeforeEachIsExecuted() {
-        FunctionalTests_Configuration_AfterEachWasExecuted = false
-
-        qck_runSpec(Configuration_AfterEachAsyncSpec.self)
-        XCTAssert(FunctionalTests_Configuration_AfterEachWasExecuted)
-
-        FunctionalTests_Configuration_AfterEachWasExecuted = false
+class AsyncConfigurationSpec: AsyncSpec {
+    override class func spec() {
+        it("should be called") {}
     }
 }

[younata]

Yeah. This is unsurprising, but still disappointing. It's also a bug with Quick, not Nimble - you'd get similar behavior if you tried the following code:

class AsyncConfiguration: QuickConfiguration {
    override class func configuration(_ configuration: QCKConfiguration) {
        configuration.beforeEach {
            let start = Date()
            RunLoop.main.run(until: Date(timeIntervalSinceNow: 2))
            expect(date.timeIntervalSince(start)).to(beCloseTo(2, within: 0.01)) // this assertion will fail when run with an async spec.
        }
    }
}

AsyncExample will run the stuff in your QuickConfiguration, as a matter of fact. This is desired because using any kind of waiting behavior in that kind of global configuration is an edge case (QuickConfiguration is meant more for doing stuff like configuring your test suite, not waiting for stuff to be available).

This is an issue when you try to run the sync versions of waitUntil, toEventually, etc. because those use APIs which no-op when run in an async context.

Possibly we might be able to get around this by detecting these cases and run them on the main actor using MainActor.run, similar to what we do for the suite hooks (i.e. the beforeSuite and afterSuite things) (see https://github.com/Quick/Quick/blob/main/Sources/Quick/Examples/AsyncExample.swift#L38-L40).