Sky Test Foundation (SkyTF) iOS

Presented live on stage at Swift Heroes 2022 Torino. Presentation deck.

Sky Test Foundation defines a domain specific language to facilitate developers writing automatic tests.

It's meant to be mobile app tests' lingua franca. Out of the box, it allows you to port tests between iOS and Android by simply copy-pasting Swift to Kotlin or vice-versa. Sky Test Foundation for Android is still in progress. The DSL allows you to define:

• http responses received by the app
• a sequence of user gestures

during test execution.

Terminology

• UX = User Experience
• SUT = System Under Test
• MA = Mobile App
• BE = Backend

Adopted Test Technique

Sky Test Foundation adopts BlackBox test technique. In general, BlackBox test technique does not require specific knowledge of the application's code, internal structure and/or programming knowledge. MA is seen as a black box as illustrated below:

MA output depends on: - user activity (user gestures) - BE state (BE http responses) - MA storage (Persistence Storage)

Outputs are:

• UI elements displayed to the user
• HTTP requests executed by the app

Tests verify the correctness of MA's behaviour defining asserts on Black Box's inputs and/or outputs.

During test execution, SkyTF allows you to:

• mock HTTP responses received by the App. You can also make assertions on each HTTP request sent by the app
• assert UI elements existence in view hierarchy
• simulate user gestures

Usage

Extend SkyUITestCase for UI tests and SkyUnitTestCase for Unit test cases.

SkyUnitTestCase - Unit Test example with SUT performing Http Requests

The goal of this kind of unit tests is to verify the correctness of the http requests performed by the MA. Using httpServerBuilder you can define the exact mock server's state during test execution, as a set of http routes.

Note: .replaceHostnameWithLocalhost() in setUp() is needed to forward http request performed by MA to the local mock server running on localhost.

import XCTest
import SkyTestFoundation
import RxBlocking
import PetStoreSDK
import PetStoreSDKTests
@testable import PetStoreApp

class LoginAPITests: SkyUnitTestCase {

    var sut: Services?

    override func setUp() {
        super.setUp()
        sut = Services(baseUrl: Urls.baseUrl().replaceHostnameWithLocalhost())
    }

    func testLogin() async throws {
        // Given
        var loginCallCount = 0
        let apiResponse = ApiResponse.mock(code: 200)
        
        httpServerBuilder.route(Routes.User.login().path) { request, callCount in
            loginCallCount = callCount
            assertEquals(request.queryParam("username"), "Alessandro")
            assertEquals(request.queryParam("password"), "Secret")
            return HttpResponse(body: apiResponse.encoded())
        }.onUnexpected{ httpRequest in
            assertFail("Unexpected http request: \(httpRequest)")
        }
        .buildAndStart()
        
        // When
        let pets = try await sut!.user.loginUser(username: "Alessandro", password: "Secret").value
    
        // Then
        assertNotNull(pets)
        assertEquals(loginCallCount, 1)
    }
}

Basic test structure:

• Given. Here you define your initial state on HTTP server mocks. In this case we defined a login route.
• When. Here you call all the methods to be tested. In this case we called the loginUser method.
• Then. Here you write all the assertions. In this case we checked pets is not nil and made sure we called login only once.

If the method under test performs an http request not handled by the mock server, then onUnexpected's closure (HttpRequest) -> () is called.

Note:

• Routes.User.login().path is a relative path not containing 127.0.0.1:8080

See the mobile app located in folder example for more details.

SkyUITestCase - PetStore App Example

Suppose we have the following user story:

As User
I want to login 
So that 
I can see a list of available pets

More details of the user story are illustrated in the following picture

And finally let's test with the help of SkyTF's DSL.

class PetListTests: SkyUITestCase {

    func testDisplayPetListView() {

          // Given
          let tom = Pet.mock(name: "Tom")
          let jerry = Pet.mock(name: "Jerry")
          let pets = [jerry, tom]

          httpServerBuilder
              .route(MockResponses.User.successLogin())
              .route(MockResponses.Pet.findByStatus(pets: pets))
              .buildAndStart()

          // When
          appLaunch()

          typeText(withTextInput("Username"), "Alessandro")
          typeText(withTextInput("Password"), "Secret")
          tap(withButton(“Login"))

          // Then
          exist(withTextEquals(tom.name))
          exist(withTextEquals(jerry.name))
    }
}

In the "Given" section we defined http mock responses required by the app, in the "When" section the app is launched and the "Login" button is tapped after user's credentials are typed. Finally in the "Then" section we assert the existence in the view hierarchy of two pets returned by the mock server. Now suppose we'd like to prove implementation's correctness of the following

As User
I want to type invalid credentials
So that 
I can see an alert "Invalid Credentials"

The associated test ca be written:

  func testLoginGivenUnauthorized() {
        // Given
        httpServerBuilder
            .route(MockResponses.User.unauthorizedLogin())
            .buildAndStart()

        // When
        appLaunch()
        exist(withTextEquals("Please login"))
        typeText(withTextInput("Username"), "Alessandro")
        typeText(withTextInput("Password"), "WrongPassword")
        tap(withButton("Login"))
        
        // Then
        exist(withTextEquals("Invalid Credentials"))
        tap(withButton("OK"))
        exist(withTextEquals("Please login"))
    }

Mock Server Builders

SkyUITestCase and SkyUnitTestCase provide mock server builder to easy the definition of the mock server routes. Builder can be accessed using the variable httpServerBuilder defined in SkyUITestCase and SkyUnitTestCase.

API - UI mock server builder

Available methods of httpServerBuilder:

public func route(_ route: HttpRoute, on: ((HttpRequest) -> Void)? = nil) -> UITestHttpServerBuilder

Adds http route to mock server. Clousure on is called on main the thread when a http request with path equals to endpoint is received by the mock server.

public func route(endpoint: HttpEndpoint, on: @escaping ((HttpRequest) -> HttpResponse)) -> UITestHttpServerBuilder

Adds http route to mock server. Closure on is called on a background thread when a http request with path equals to endpoint is received by the mock server. The closure allows to define different Http responses given the same endpoint.

func buildAndStart(port: in_port_t = 8080, file: StaticString = #file, line: UInt = #line) throws -> HttpServer

Build all routes added so far and starts the mock server.

public func callReport() -> [EndpointReport]

Returns a report of defined of routes. See EndpointReport for more details.

public func undefinedRoute(_ asserts: @escaping (HttpRequest) -> Void) -> UITestHttpServerBuilder

It allows to define assert on http requests not handled by the mock server.

public func routeImagesAt(path: String, properties: ((HttpRequest) -> ImageProperties)? = nil) -> UITestHttpServerBuilder {

It allows to define endpoint returning image dynamically create by the server.

Example

import XCTest
import Swifter
import SkyTestFoundation
class UITests: SkyUITestCase {

    func test() throws {
        // Given
        httpServerBuilder
            .route(endpoint: HttpEndpoint("/endpoint1"), on: { (request) -> HttpResponse in
                return HttpResponse(body: Data())
            })
            .buildAndStart()

        appLaunched()
        // ...
    }
}

The test is composed by 3 sections:

• Given: mocks, http routes are defined and app is launched
• When: ui gesture are performed in order to navigate to the view to be tested
• Then: assertions on ui element of the view (to be tested)

DSL for UI Testing

SkyTestFoundation provides a simple DSL in order to facilitate the writing of UI tests. It is a thin layer defined on top of primtives offered by XCTest. The same DSL for testing is defined for Android platform on top of Espresso (see client-lib-android-test-foundation). SkyTestFoundation custom assertions are wrappers of events defined in XCUIElement like tap(). DSL assertions wait for any element to appear before firing the wrapped event. One of the effect of using custom assertions is to reduce flakiness of ui test execution.

• exist(_ element) Determines if the element exists.
• notExist(_ element) Determines if the element NOT exists.
• tap(_ element) Sends a tap event to a hittable point computed for the element.
• doubleTap(_ element) Sends a double tap event to a hittable point computed for the element.
• isEnabled(_ element) Determines if the element is enabled for user interaction.
• isNotEnabled(_ element) Determines if the element is NOT enabled for user interaction.
• isRunningOnSimulator() -> Bool Returns true if ui test is running on iOS simulator. It can be used in conjunction with XCTSkipIf/1 in order to skip the execution of a ui test if on iOS simulator.
• withTextEquals(_ text) A XCUIElementQuery query for locating staticText view elements equals to text
• withTextContains(_ text) A XCUIElementQuery query for locating staticText view elements containing text
• withIndex(_ query, index) the index-th element of the result of the query query
• assertViewCount(_ query, expectedCount) Asserts if the number of view matched by query is equals to expectedCount
• swipeUp(_ element) performs swipe up user gesture on element
• swipeDown(_ element) performs swipe up user gesture on element
• swipeLeft(_ element) performs swipe up user gesture on element
• swipeRight(_ element) performs swipe up user gesture on element
• swipeUp() performs swipe up user gesture
• swipeDown() performs swipe down user gesture
• swipeLeft() performs swipe left user gesture
• swipeRight() performs swipe right user gesture

Notice: DSL for testing allows to write iOS UI Test and copy it to android and viceversa.

Mocks - Random data generators

The framework provides mocks for built-in data types of Swift. In mock testing, the dependencies are replaced with objects that simulate the behavior of the real ones. The purpose of mocking is to isolate and focus on the code being tested and not on the behavior or state of external dependencies. Each mocks returns a random value of the associated data type.

Example

var v = String.mock()
print(v)  // prints 673D6E7C-ECE4-493C-B86B-25DAE78C02CC
v = String.mock()
print(v)  // prints 2D092A17-5BBB-4F91-8E4B-BC45A902D235

Real Data / Dictionaries

Real data dictionaries can be used to assign meaningful values to generated mocks. Available real data dictionaries are:

Example

var v = String.mock(.firstname) // randomly generate a firstname value
print(v) // prints Augusto
v = String.mock(.firstname)
print(v) // prints Elisa

Installation

Swift Package Manager

SPM is supported

Demos

Source code available at: https://github.com/sky-uk/client-lib-ios-test-foundation/tree/demos

Demo iOS App

The app requests a text and an image to the mock sever. The project includes an UI test example showing mock server usage.

import XCTest
import SkyTestFoundation

class DemoIOSUITests: SkyUITestCase {

    func testMockServer() throws {
        // Given
        let text = "Hello world from SkyTestFoundation Mock Server."
        try httpServerBuilder
            .routeImagesAt(path: "/image", properties: nil)
            .route((endpoint: "/message", statusCode: 200, body: text.data(using: .utf8)!, responseTime: 0))
            .buildAndStart()
        // When
        let app = XCUIApplication()
        app.launch()
        // Then
        exist(app.staticTexts[text])
        exist(app.windows
                .children(matching: .other).element
                .children(matching: .other).element
                .children(matching: .other).element
                .children(matching: .image).element)

        httpServerBuilder.httpServer.stop()
    }
}

The following view will be displayed in the iOS simulator during the test execution:

Demo MacOS App

The same test of Demo iOS App is executued.

Note: pay attention to settings/capabilities of target app, in order to perform http request to localhost from the app, and entitlements set to UI test target in order to allow socket bind to localhost.

The following view is displayed during the execution of the test:

List of acronyms

• MA mobile iOS application
• SUT system under test

Examples

Unit Test and callCount

callCount stores the number of http request call received by the mock server for a specific endpoint.

 func testCallCountExample() throws {
     let exp00 = expectation(description: "expectation 00")
     
     var callCount0 = 0
     var callCount1 = 0
     httpServerBuilder
       .route("/endpoint/1") { (request, callCount) -> (HttpResponse) in
           callCount0 = callCount
           return HttpResponse(body: Data())
        }
        .route("/endpoint/2") { (request, callCount) -> (HttpResponse) in
            callCount1 = callCount
            return HttpResponse(body: Data())
        }
        .buildAndStart()

        let session = URLSession(configuration: URLSessionConfiguration.default)

        let url00 = URL(string: "http://localhost:8080/endpoint/1")!
        let dataTask00 = session.dataTask(with: url00) { (_, _, error) in
            XCTAssertNil(error)
            exp00.fulfill()
        }

        dataTask00.resume()

    wait(for: [exp00], timeout: 3)
    XCTAssertEqual(callCount0, 1)
    XCTAssertEqual(callCount1, 0)
}