Fixed `tvOS` tests #1928

NachoSoto · 2022-09-21T19:58:32Z

Turns out we hadn't been running tests on tvOS for a while:

[19:08:22]: Couldn't find any matching simulators for '["Apple TV (15.2)"]' - falling back to default simulator
[19:08:22]: Found simulator "iPhone 13 Pro (15.5)"

I found this went I went to dig into this failure. Turns out that wasn't running on tvOS anyway.

This turns them back on, fixes the compilation issues on tvOS tests, and ensures that we don't run into this issue again.

Changes:

Renamed device in lane test_tvos
Enabled ensure_devices_found on iOS and tvOS lanes. This will also ensure that we don't stop running tests on iOS XX because it doesn't find that simulator.
Removed UnitTestHostApp storyboards and scene, since those aren't supported in tvOS, and aren't needed because there's no UI.
Made StoreKitUnitTests and UnitTestHostApp targets universal
No longer linking AdServices on tvOS in test targets
Disabled attribution tests on tvOS. Those methods weren't being compiled, but we were testing that a message was being sent, even though it no longer was.
Changed iOS 14 tests to use Xcode 13 due to Xcode 14 simulator list blank xcpretty/xcode-install#466 (same as CircleCI: changed iOS 12/13 to use Xcode 13 #1918).

NachoSoto · 2022-09-21T20:04:54Z

fastlane/Fastfile

@@ -121,6 +121,7 @@ platform :ios do
    scan(
      step_name: "scan - iPhone",
      device: ENV['SCAN_DEVICE'] || "iPhone 12 (15.2)",
+      ensure_devices_found: true,


Thanks for the knowledge @joshdholtz!

Turns out we hadn't been running tests on `tvOS` for a while: > [19:08:22]: Couldn't find any matching simulators for '["Apple TV (15.2)"]' - falling back to default simulator > [19:08:22]: Found simulator "iPhone 13 Pro (15.5)" This turns them back on, fixes the compilation issues on `tvOS` tests, and ensures that we don't run into this issue again. ### Changes: - Renamed device in lane `test_tvos` - Enabled `ensure_devices_found` on `iOS` and `tvOS` lanes. This will also ensure that we don't stop running tests on iOS XX because it doesn't find that simulator. - Removed `UnitTestHostApp` storyboards and scene, since those aren't supported in `tvOS`, and aren't needed because there's no UI. - Made `StoreKitUnitTests` and `UnitTestHostApp` targets universal - No longer linking `AdServices` on `tvOS` in test targets - Disabled attribution tests on `tvOS`. Those methods weren't being compiled, but we were testing that a message was being sent, even though it no longer was.

NachoSoto · 2022-09-21T20:08:21Z

.circleci/config.yml

+      - install-and-create-sim:
+          install-name: iOS 14.5 Simulator
+          sim-device-type: iPhone-8
+          sim-device-runtime: iOS-14-5
+          sim-name: iPhone 8 (14.5)


This was missing now in Xcode 14.0

NachoSoto · 2022-09-21T20:08:29Z

.circleci/config.yml

@@ -235,7 +235,7 @@ jobs:
          command: bundle exec fastlane test_ios
          no_output_timeout: 30m
          environment:
-            SCAN_DEVICE: iPhone 13 (16.0)
+            SCAN_DEVICE: iPhone 14 (16.0)


This was wrong too.

joshdholtz

Just one question on deleted tests but approving because I assume there is a good reason 😊

joshdholtz · 2022-09-23T00:36:59Z

Tests/StoreKitUnitTests/BeginRefundRequestHelperTests.swift

-    func testBeginRefundRequestForProductFatalErrorIfNotIosOrCatalyst() throws {
-        #if os(watchOS) || os(macOS) || os(tvOS)
-        expectFatalError(expectedMessage: Strings.purchase.begin_refund_request_unsupported.description) {
-            _ = try await helper.beginRefundRequest(forProduct: mockProductID)
-        }
-        #else
-        throw XCTSkip("beginRefundRequestForProduct is not available for this test.")
-        #endif
-    }
-
-    func testBeginRefundRequestForEntitlementFatalErrorIfNotIosOrCatalyst() throws {
-        #if os(watchOS) || os(macOS) || os(tvOS)
-        expectFatalError(expectedMessage: Strings.purchase.begin_refund_request_unsupported.description) {
-            _ = try await helper.beginRefundRequest(forEntitlement: mockEntitlementID)
-        }
-        #else
-        throw XCTSkip("beginRefundRequestForEntitlement is not available for this test.")
-        #endif
-    }
-
-    func testBeginRefundRequestForActiveEntitlementFatalErrorIfNotIosOrCatalyst() throws {
-        #if os(watchOS) || os(macOS) || os(tvOS)
-        expectFatalError(expectedMessage: Strings.purchase.begin_refund_request_unsupported.description) {
-            _ = try await helper.beginRefundRequestforActiveEntitlement()
-        }
-        #else
-        throw XCTSkip("beginRefundRequestForEntitlement is not available for this test.")
-        #endif
-    }
-
-#if os(iOS)


Why are these being deleted? 🤔

Oh I forgot to explain. The expectation wasn't being met (maybe it was before?). The methods aren't even being compiled anymore.

Literally no code used that message string

Turns out we hadn't been running tests on `tvOS` for a while: > [19:08:22]: Couldn't find any matching simulators for '["Apple TV (15.2)"]' - falling back to default simulator > [19:08:22]: Found simulator "iPhone 13 Pro (15.5)" I found this went I went to dig into [this failure](#1926 (comment)). Turns out that wasn't running on `tvOS` anyway. This turns them back on, fixes the compilation issues on `tvOS` tests, and ensures that we don't run into this issue again. ### Changes: - Renamed device in lane `test_tvos` - Enabled `ensure_devices_found` on `iOS` and `tvOS` lanes. This will also ensure that we don't stop running tests on iOS XX because it doesn't find that simulator. - Removed `UnitTestHostApp` storyboards and scene, since those aren't supported in `tvOS`, and aren't needed because there's no UI. - Made `StoreKitUnitTests` and `UnitTestHostApp` targets universal - No longer linking `AdServices` on `tvOS` in test targets - Disabled attribution tests on `tvOS`. Those methods weren't being compiled, but we were testing that a message was being sent, even though it no longer was. - Changed iOS 14 tests to use Xcode 13 due to xcpretty/xcode-install#466 (same as #1918).

I blindly copy-pasted this snapshot and forgot that iOS 12 has different formatting. This was wrong in #1926. It wasn't caught originally because we hadn't been running tests on iOS 12 until #1928.

@NachoSoto

**This is an automatic release.** ### Bugfixes * `Purchases.beginRefundRequest`: ensured errors are `PublicError` (#1913) via NachoSoto (@NachoSoto) * `PurchaseTesterSwiftUI`: fixed macOS target (#1915) via NachoSoto (@NachoSoto) ### Other Changes * `SnapshotTesting`: require version 1.9.0 to keep supporting iOS 12/13 tests (#1931) via NachoSoto (@NachoSoto) * Fixed `tvOS` tests (#1928) via NachoSoto (@NachoSoto) * `pre-commit` hook: also verify leftover API keys in `PurchaseTester` (#1914) via NachoSoto (@NachoSoto) * `CircleCI`: changed iOS 12/13 to use Xcode 13 (#1918) via NachoSoto (@NachoSoto) * `PurchaseTesterSwiftUI`: removed unnecessary `UIApplicationDelegate` (#1916) via NachoSoto (@NachoSoto) * `CircleCI`: changed all jobs to use Xcode 14 (#1909) via NachoSoto (@NachoSoto) * `Atomic`: added unit test to verify `value`'s setter (#1905) via NachoSoto (@NachoSoto) * `spm build` CI job: changed to release build (#1903) via NachoSoto (@NachoSoto) * `StoreKitUnitTests`: compile on iOS 11.0+ (#1904) via NachoSoto (@NachoSoto) * `Purchases`: only expose testing data on `DEBUG` (#1902) via NachoSoto (@NachoSoto) * `Integration Tests`: added test to verify re-subscription behavior (#1898) via NachoSoto (@NachoSoto) * `IntegrationTests`: simplified `testExpireSubscription` to fix flaky test (#1899) via NachoSoto (@NachoSoto) * `Integration Tests`: actually verify that entitlement is active (#1880) via NachoSoto (@NachoSoto)

NachoSoto added test labels Sep 21, 2022

NachoSoto requested a review from a team September 21, 2022 19:58

NachoSoto commented Sep 21, 2022

View reviewed changes

NachoSoto added 3 commits September 21, 2022 13:05

Also install iOS 14 simulator

3ce88e8

Fixed iOS 16 simulator

c4e91c4

NachoSoto force-pushed the fix-tvos-tests branch from db5c981 to c4e91c4 Compare September 21, 2022 20:08

NachoSoto commented Sep 21, 2022

View reviewed changes

NachoSoto added 3 commits September 21, 2022 13:23

Also print simulators

59785d6

Use Xcode 13 for iOS 14

0b05e00

No need to install simulator anymore

cb37c4c

NachoSoto mentioned this pull request Sep 22, 2022

CacheableNetworkOperation: fixed race condition in new test #1932

Merged

joshdholtz approved these changes Sep 23, 2022

View reviewed changes

NachoSoto merged commit e3c0dfe into main Sep 23, 2022

NachoSoto deleted the fix-tvos-tests branch September 23, 2022 17:32

NachoSoto mentioned this pull request Sep 23, 2022

[AUTOMATIC] Release/4.12.1 #1923

Merged

NachoSoto mentioned this pull request Sep 23, 2022

Fixed iOS 12 tests #1936

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fixed `tvOS` tests #1928

Fixed `tvOS` tests #1928

NachoSoto commented Sep 21, 2022 •

edited

Loading

NachoSoto Sep 21, 2022

NachoSoto Sep 21, 2022

NachoSoto Sep 21, 2022

joshdholtz left a comment

joshdholtz Sep 23, 2022

NachoSoto Sep 23, 2022

NachoSoto Sep 23, 2022

Fixed tvOS tests #1928

Fixed tvOS tests #1928

Conversation

NachoSoto commented Sep 21, 2022 • edited Loading

Changes:

NachoSoto Sep 21, 2022

Choose a reason for hiding this comment

NachoSoto Sep 21, 2022

Choose a reason for hiding this comment

NachoSoto Sep 21, 2022

Choose a reason for hiding this comment

joshdholtz left a comment

Choose a reason for hiding this comment

joshdholtz Sep 23, 2022

Choose a reason for hiding this comment

NachoSoto Sep 23, 2022

Choose a reason for hiding this comment

NachoSoto Sep 23, 2022

Choose a reason for hiding this comment

Fixed `tvOS` tests #1928

Fixed `tvOS` tests #1928

NachoSoto commented Sep 21, 2022 •

edited

Loading