Unable to use when tests don't have access to sources (Xcode Cloud)

[cameroncooke]

When attempting to run snapshot tests on a workflow where the tests are run in a different environment to the where the tests are built (sources unavailable) assertSnapshot(...) is unable to find the snapshots using the current logic.

A concrete example of this setup is when using Xcode Cloud CI service which uses one environment to build the tests and then another environment to execute the tests. On the environment where the tests are executed, there is no access to the build sources that the StaticString = #file resolves to at build-time.

While Xcode Cloud scripts can be used to copy files to a single ci_scripts directory on the environment, when running the tests there is no way to override the path used by assertSnapshot(...) at run-time to point it to the directory of reference images because of the use of StaticString.

It might make sense to have a way of overriding the business rules around where the snapshots files are located for a given test so that it can be used in situations where we can't rely on the automatic behaviour.

More information on why test actions don't have access to sources can be found here: https://developer.apple.com/videos/play/wwdc2021/10269/?time=711

[cameroncooke]

Would it be possible to allow the use of run-time test path strings in assertSnapshot so we can use it in constrained environments where the current opinionated path logic doesn't work?

[dave-perry]

Also having this problem, a solution would be very much appreciated!

[qmoya]

Our project is structured à la isowords, so we could work around the issue in this fashion: with a custom assertion that deduces the path to ci_scripts and outputs the snapshots inside of it.

[cameroncooke]

I like @qmoya's solution and I think it's the best option sans changes to this library but it does mean we have to duplicate the assertSnapshot function with our own specific version and then maintain this if and when this library's API evolves. Maybe a strategy based API for controlling the snapshot directory/naming logic could work well here.

[mortenbekditlevsen]

I experimented a bit with Xcode Cloud and ci_scripts - and I'll add my findings here - in case it can help someone in the future.

I read that only contents within ci_scripts would be available for test runs, so my first idea was for the ci_post_clone.sh script to rsync all directories with the name __Snapshots__ with their relative path into ci_scripts.
This approach does not work, since the files that are copied into ci_scripts in post clone are not available when testing. It appears that a clean version of the ci_scripts folder is made available instead.

My second attempt was to create symbolic links to the __Snapshots__ folders from relative paths like ci_scripts/Modules/Tests/ReportTesting/__Snapshots__ - since the documentation states that symbolic links present in the ci_scripts folder is copied out before testing.
This approach also does not work since it would appear that only symbolic links present directly in the root of ci_scripts are copied over.

My final approach is to have all my snapshots be placed under an Artifacts folder in the root of my repository - otherwise keeping the folder structure that is present when using the default assertSnapshot.
I create a symbolic link from ci_scripts/Artifacts to Artifacts and this copies them over nicely.
In my assertSnapshot helper I detect if we're running on CI and append ci_scripts to the path that is used.

And then instead of importing SnapshotTestcase I just import my own TestHelpers and everything runs locally and on Xcode Cloud.

Here's my assertSnapshot implementation:

//
//  AssertSnapshot.swift
//  
//
//  Created by Morten Bek Ditlevsen on 23/12/2021.
//

import Foundation
import SnapshotTesting
import XCTest

public func assertSnapshot<Value, Format>(
    matching value: @autoclosure () throws -> Value,
    as snapshotting: Snapshotting<Value, Format>,
    named name: String? = nil,
    record recording: Bool = false,
    timeout: TimeInterval = 5,
    file: StaticString = #file,
    testName: String = #function,
    line: UInt = #line
) {
    let isCI = ProcessInfo.processInfo.environment["CI"] == "TRUE"
    let sourceRoot = URL(fileURLWithPath: ProcessInfo.processInfo.environment["SOURCE_ROOT"]!,
                         isDirectory: true)

    let fileUrl = URL(fileURLWithPath: "\(file)", isDirectory: false)
    let fileName = fileUrl.deletingPathExtension().lastPathComponent

    let absoluteSourceTestPath = fileUrl
        .deletingLastPathComponent()
        .appendingPathComponent("__Snapshots__")
        .appendingPathComponent(fileName)
    var components = absoluteSourceTestPath.pathComponents
    let sourceRootComponents = sourceRoot.pathComponents
    for component in sourceRootComponents {
        if components.first == component {
            components = Array(components.dropFirst())
        } else {
            XCTFail("Test file does not share a prefix path with SOURCE_ROOT")
            return
        }
    }
    var snapshotDirectoryUrl = sourceRoot
    if isCI {
        snapshotDirectoryUrl = snapshotDirectoryUrl.appendingPathComponent("ci_scripts")
    }
    snapshotDirectoryUrl = snapshotDirectoryUrl.appendingPathComponent("Artifacts")
    for component in components {
        snapshotDirectoryUrl = snapshotDirectoryUrl.appendingPathComponent(component)
    }

    let failure = verifySnapshot(
        matching: try value(),
        as: snapshotting,
        named: name,
        record: recording,
        snapshotDirectory: snapshotDirectoryUrl.path,
        timeout: timeout,
        file: file,
        testName: testName
    )
    guard let message = failure else { return }
    XCTFail("\(message) snap: \(snapshotDirectoryUrl)", file: file, line: line)
}

One thing that I'm unsure about is how the environment propagates to the tests, so in my test scheme I had to manually forward CI as $(CI) and SOURCE_ROOT as $(SOURCE_ROOT).
I'm suspecting that there's actually a better way to handle the environment, but it works for now...

[PorterHoskins]

I can't figure out how to get this to work. I even tried putting the files directly in ci_scripts but the test step in Xcode Cloud does not carry over the files for me.

[oliverfoggin]

This seems to fail on the line

let sourceRoot = URL(fileURLWithPath: ProcessInfo.processInfo.environment["SOURCE_ROOT"]!,
                         isDirectory: true)

Unexpectedly found nil.

[michallaskowski]

Xcode Cloud is open now. Tried to use it, and that's how I have found this discussion. While some workarounds were probably ok when it was in beta, with bigger adoption it would be great to add some support to ease the pain of using snapshot tests on CI.
While having the script that copies the snapshots to ci_scripts folder seems inevitable, custom func assertSnapshot in every project using Xcode Cloud seems a little excessive.
Could we maybe check for an env variable that contains the snapshots root path, in the library?
Happy to open a PR.

[cameroncooke]

You don't need to copy snapshots. You can use symbolic links. Xcode Cloud copies the linked files when copying the ci_scripts/ directory to other build instances. We symbolic link the entire snapshots directory into the ci_scripts/ directory. Then update our snapshot tests to use the correct path when running tests on CI environment.

[michallaskowski]

Do you have a script that creates the symlink in ci_post_clone.sh, or just a symlink that is checked into the git repo? If the latter that would be great.
But updating the library to handle that case better still stands ;) (unless there is sth else I am not aware yet)

[cameroncooke]

Symlinks are checked in, it’s a relative link so is able to be linked to the snapshots directory. I agree this library is very opinionated on where the files should be located.

On Thu, 9 Jun 2022 at 13:57, Michal Laskowski ***@***.***> wrote: Do you have a script that copies the files in ci_post_clone.sh, or just a symlink that is checked into the directory? If the latter that would be great. But updating the library to handle that case better still stands ;) (unless there is sth else I am not aware yet) — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SIHJXKFGQPUHAUKNZDVOHS4ZANCNFSM5KSOPA2A> . You are receiving this because you authored the thread.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/2913436@ github.com>

-- Cameron.

[cameroncooke]

The ci_scripts directories and all sub directories are copied (following any symlinks) to each test instance. We’ve been doing this for nearly a year now and works fine for us. The thing you have to do is feed the snapshot function an invalid file path so that when the path elements get removed at runtime it ends up being the correct file path to the ci_scripts directory. It’s a bit of trial and error to get right hence opening this issue. See https://developer.apple.com/documentation/xcode/writing-custom-build-scripts

On Fri, 17 Jun 2022 at 20:15, Porter ***@***.***> wrote: I can't figure out how to get this to work. I even tried putting the files directly in ci_scripts but the test step in Xcode Cloud does not carry over the files for me. — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SP55XXNPZ6A3N5SHHLVPTFEFANCNFSM5KSOPA2A> . You are receiving this because you authored the thread.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/2973898@ github.com>

-- Cameron.

[armintelker]

Hey, can you share the code that worked for you?
the ci script copy/linking part and the adaption of the snapshot tests itself.
Thanks :)

[ghost]

Hi @mortenbekditlevsen @qmoya ,
Copied all *.png into related path under ci_scripts/
The test still fail on Xcode Cloud but worked on local, so I can't figure out how workaround.

private func snapshotDirectory(
    for file: StaticString,
    anchorRepoPathComponent: String = "Modules",
    ciScriptsPathComponent: String = "ci_scripts",
    snapshotsPathComponent: String = "__Snapshots__"
) -> String {
    
    let fileURL = URL(fileURLWithPath: "\(file)", isDirectory: false)
    
    let packageRootURL: URL = {
        var newValue = fileURL
        while newValue.lastPathComponent != anchorRepoPathComponent {
            newValue = newValue.deletingLastPathComponent()
        }
        return newValue.deletingLastPathComponent()
    }()
    
    let testPathComponent = fileURL
        .deletingPathExtension()
        .lastPathComponent
    
    let snapshotDirectoryPath = packageRootURL
        .appendingPathComponent(ciScriptsPathComponent)
        .appendingPathComponent(snapshotsPathComponent)
        .appendingPathComponent(testPathComponent)
        .path
    
    print("snapshotDirectoryPath: \(snapshotDirectoryPath)")
    return snapshotDirectoryPath
}

[alfonsogarza]

I was able to follow @cameroncooke comments and replicate on my side.

No need for a custom ci_script, just create a symlink from ci_scripts to your screenshot destination - Xcode Cloud even copies recursive folders.

Then use verifySnapshot passing the directory depending on whether it is local or CI and assert (like the custom assertSnapshot methods above)

      if ProcessInfo.processInfo.environment["CI"] == "TRUE" {
            return URL(fileURLWithPath: "\(#file)", isDirectory: false)
                .deletingLastPathComponent()
                .deletingLastPathComponent()
                .deletingLastPathComponent()
                .appendingPathComponent("ci_scripts")
                .appendingPathComponent("__Snapshots__")
        }

        return URL(fileURLWithPath: "\(#file)", isDirectory: false)
            .deletingLastPathComponent()
            .appendingPathComponent("__Snapshots__")

The number of .deletingLastPathComponent() really depends on where your folder is. If you don't know where yours is do a run with XCTFail(#file) and use that as reference to find how many removals to get to .../repository/

To get ProcessInfo.processInfo.environment["CI"] == "TRUE" to work I had to make sure to pipe it through the scheme environment variables.

[broomburgo]

I was able to make this work in a way similar to the proposed solutions, that is, by manually specifying the snapshot path in the verifySnapshot function, but I'm now seeing another weird behavior:

• if I run an Xcode Cloud workflow that has only a Test action, it works as expected (snapshots are found);
• if I run an Xcode Cloud workflow that has both Test and Archive actions the snapshots are not found, and it looks exactly like before.

Did anybody encounter a similar problem?

[oliverfoggin]

Hey, did you manage to resolve this? Right now I can't get any snapshot tests to work. 😅

Thanks

[oliverfoggin]

Hmm... struggling with this still...

I've tried @qmoya and @mortenbekditlevsen solutions but they both seem to be missing something.

I'll keep going 👍🏻

[lucasecf]

I'm struggling right now as well - I did the symlink + custom method with verifySnapshot passing the symlink path. Locally it works fine, but on Xcode Cloud the test actually hangs for some reason 🤔 . Please share if you can make it work

[oliverfoggin]

For me so far I've got it working locally but on Xcode Cloud it can't find the snapshots so records them again.

[LukeDurrant]

I use this at the moment.
https://gist.github.com/LukeDurrant/2793281adca7411fb6df6c1db1ee091b
symlink I couldn't get to work correctly with a simple flow so now have all my snapshots in the ci_scripts directory. The above gist keeps the path structure correct so it isn't too bad.

[LukeDurrant]

Side note.
Assessing Xcode Cloud over the last few months.
As product its expensive and support is terrible.
It took forever to get any support (I asked to upgrade my minutes as it was US summer but no info on when and find out why non of my builds where being started the issue fixed itself after 48 hours, but no incident reporting on status)
I'll likely just keep it for creating releases and run tests else where.

[oliverfoggin]

Thanks, do you have to use any additional ci_scripts or anything? Or do you just use this to set the directory? Thanks

[oliverfoggin]

Do you have any additional files or scripts needed to make this work? At the moment this doesn't seem to work on its own.

[cameroncooke]

We use swift-snapshot-testing without any changes to the library code using Xcode Cloud as I mentioned above just using symlinks and setting up our tests to inject a different path when running on CI to theassertSnapshot(file: StaticString) argument. How you detect CI is entirely up to you, can be done with a custom scheme and environmental variables or you can just inspect the filesystem.

Because assertSnapshot() manipulates the path you have to inject a path that once modified by the library ends up being the real logical path. This means the path you inject when running on CI won't actually be a valid path on the file-system so you should structure your path in such a way so that it ends up being correct.

Here is a basic example of what we're doing, though this can of course be optimised:

final class ViewTests: XCTest {
    
    let ciPath: StaticString = "/Volumes/workspace/repository/ci_scripts/SnapshotTests/ViewTests.swift"
    let localPath: StaticString = #file
    var isCIEnvironment: Bool { ... }

    func testSomething() {
        let view = View()

        var filePath: StaticString
        if isCIEnvironment {
            filePath = ciPath
        } else {
            filePath = localPath
        }

        assertSnapshot(
            matching: view,
            file: filePath
        )    
    }
}

The key here is these two lines:

    let ciPath: StaticString = "/Volumes/workspace/repository/ci_scripts/resources/SnapshotTests/ViewTests.swift"
    let localPath: StaticString = #file

We have to ensure that the ciPath once manipulated ends up being the real logical path when running on Xcode Cloud.
The below example shows the logical file-system of a made up project and symlink:

ci_scripts/
├─ resources/
│  ├─ SnapshotTests/ 
|  │  ├─ __Snapshots__/ -> ../../../../MyProject/SnapshotTests/__Snapshots__ 
MyProject/
├─ SnapshotTests/
│  │  ├─ __Snapshots__/ <- Symlink target directory
│  │  │  ├─ ViewTests/
│  │  │  │  ├─ snapshot1.png
│  │  │  │  ├─ snapshot2.png

On your local file-system the symlinks just act as aliases to the files at the target locations but Xcode Cloud "slurps" i.e. copies the target files and directories that the symlinks resolved to to the location of the symlink itself.

If we inspected the ci_scripts directory on Xcode Cloud for a test environment it would actually look like:

ci_scripts/
├─ resources/
│  ├─ SnapshotTests/ 
|  │  ├─ __Snapshots__/
|  │  │  ├─ ViewTests/ 
│  │  │  │  ├─ snapshot1.png
│  │  │  │  ├─ snapshot2.png

Hope this helps.

[oliverfoggin]

Ah, good idea, I'll do that now. Thanks.

The thing that makes me think they're not there is Xcode telling me it can't find them there. It has the correct directory path and file name. It just tells me it can't find the file.

😅

[oliverfoggin]

OK, I added some logs in...

Here is the snapshot directory I pass into the snapshot tests...

🍎 /Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests

(I added the 🍎 to make it easier to find).

Then the error tells me...

/Volumes/workspace/repository/Tests/AppFeatureTests/AppCoreTests.swift:12: error: -[AppFeatureTests.AppCoreTests testAnalytics] : failed - No reference was found on disk. Automatically recorded snapshot: …

open "/Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests/testAnalytics.1.txt"

So the directory it has created the testAnalytics.1.txt and the directory I pointed it at are identical.

[oliverfoggin]

OK...

The single test function does multiple snapshot tests so each snapshot file is suffixed with a number .1, .2, .3, etc...

During the first test...

2022-10-05T18:27:28.042437930Z	🍏 /Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests
2022-10-05T18:27:28.042581615Z	🍏 ["No file found"]

Then on the next test...

2022-10-05T18:27:43.094693048Z	🍏 /Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests
2022-10-05T18:27:43.094902904Z	🍏 ["testAnalytics.1.txt"]

Then...

2022-10-05T18:27:43.097634127Z	🍏 ["testAnalytics.1.txt", "testAnalytics.2.txt"]
2022-10-05T18:27:43.099620080Z	🍏 ["testAnalytics.1.txt", "testAnalytics.2.txt", "testAnalytics.3.txt"]
2022-10-05T18:27:43.101803866Z	🍏 ["testAnalytics.1.txt", "testAnalytics.2.txt", "testAnalytics.3.txt", "testAnalytics.4.txt"]

And so on.

So... it's definitely able to add snapshot files to the same place that I'm expecting them to be.

From the root of my repo I'm storing these files in ...

<repo_root>/ci_scripts/resources/Tests/RecurringSavesTests/__Snapshots__/RecurringSavesCoreTests

Hmm... I'm still sticking at it. Thanks for the help with this :D

[Archdoog]

One subtle note here from my experience troubleshooting and successfully setting up this solution. The symlink is relative in the diagram. This can be confusing/tricky as the default macOS ln command does not support relative -r.

As a result, I had to use the gnu ln command bundled in the brew package coreutils. It was preinstalled with brew, but I needed the ln inside it's bin folder.

# if you need to install coreutils:
brew install coreutils

# to find the path to the coreutils bin:
brew info coreutils

# Running the relative symlink command (with my brew's coreutils path):
/opt/homebrew/opt/coreutils/libexec/gnubin/ln -rs ~/Path/to/Repo/TestTarget/__Snapshots__ ci_scripts/resources/TestTarget/__Snapshots__

You'll obviously need to adjust and play around with the paths to your repo, snapshots folder(s), etc. The key is to make sure it's a relative symlink.

[omar-delphia]

This was very helpful, thank you!

I wanted to create the file path dynamically so I could use a shared helper method. Using assertSnapshot doesn't work in that case because it needs a StaticString for the file. I landed on using verifySnapshot and pass in the snapshotDirectory instead, which can be dynamic. If anyone runs into this in the future, that would look something like this (I'm sure this can be cleaned up & I'm not sure file.description is the recommended way of getting a String from a StaticString):

Also if you have any nested folders make sure you add symlinks for them too.

func assertSnapshot<Value, Format>(
    matching value: @autoclosure () throws -> Value,
    as snapshotting: Snapshotting<Value, Format>,
    named name: String? = nil,
    record recording: Bool = false,
    timeout: TimeInterval = 5,
    file: StaticString = #file,
    testName: String = #function,
    line: UInt = #line
) {
    let ciPathPrefix = "/Volumes/workspace/repository/ci_scripts/"
    let components = URL(string: file.description)!.pathComponents
    let projectIndex = components.firstIndex(of: "MyProject")!
    let folders = components[(projectIndex + 1)..<components.endIndex - 1].joined(separator: "/")
    let fileName = (components[components.endIndex - 1] as NSString).deletingPathExtension
    let snapshotDirectory = "\(ciPathPrefix)\(folders)/__Snapshots__/\(fileName)"

    let failure = verifySnapshot(
        matching: try value(),
        as: snapshotting,
        named: name,
        record: recording,
        snapshotDirectory: snapshotDirectory,
        timeout: timeout,
        file: file,
        testName: testName,
        line: line
    )
    guard let message = failure else { return }
    XCTFail(message, file: file, line: line)
}

[cameroncooke]

So it’s not Xcode telling you they are no there it’s assertSnapshot() function’s logic that’s saying it can’t find the images so I would suspect the issue lies there as I see no reason why they wouldn’t be in the ci_snapshots/ if you put them there.

On Wed, 5 Oct 2022 at 18:22, Oliver Foggin ***@***.***> wrote: Ah, good idea, I'll do that now. Thanks. The thing that makes me think they're not there is Xcode telling me it can't find them there. It has the correct directory path and file name. It just tells me it can't find the file. 😅 — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SMTVNP6WXHNGWI7263WBW2MBANCNFSM5KSOPA2A> . You are receiving this because you were mentioned.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/3807522@ github.com>

-- Cameron.

[oliverfoggin]

OK, interesting. I'll have a look deeper at what's going on under the hood.

[cameroncooke]

The two paths look different to me one is missing the resources dir.

On Wed, 5 Oct 2022 at 19:09, Oliver Foggin ***@***.***> wrote: OK, interesting. I'll have a look deeper at what's going on under the hood. — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SMA3NL5FVQDNJMYTZDWBW74XANCNFSM5KSOPA2A> . You are receiving this because you were mentioned.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/3807888@ github.com>

-- Cameron.

[oliverfoggin]

The one that is missing the resources directory is the file path to the test file. Not the directory path of the snapshotDirectory.

[cameroncooke]

I’m not sure that is correct, when it says recording new snapshots and gives a path to the new snapshot that directory of the new snapshot is the location it was searching for and couldn’t fix the reference snapshot. So this doesn’t look correct to me. They should be the same in my opinion.

On Wed, 5 Oct 2022 at 19:11, Oliver Foggin ***@***.***> wrote: The one that is missing the resources directory is the file path to the test file. Not the directory path of the snapshotDirectory. — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SNQ3XW7YSEIAY2DQBDWBXAFRANCNFSM5KSOPA2A> . You are receiving this because you were mentioned.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/3807903@ github.com>

-- Cameron.

[oliverfoggin]

OK, I'm not sure what your referring to but the file paths are the same. One of them isn't missing the resources. 😄

[oliverfoggin]

I’m not sure that is correct, when it says recording new snapshots and
gives a path to the new snapshot that directory of the new snapshot is the
location it was searching for and couldn’t fix the reference snapshot.

The location of the new snapshot is...

/Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests/testAnalytics.1.txt

The snapshot directory I am giving it is...

/Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests

[oliverfoggin]

OK... I logged out...

	let foo = try? FileManager.default.contentsOfDirectory(atPath: "/Volumes/workspace/repository")
	print("🍏 \(foo ?? ["No file found"])")

The first snapshot test it runs logs... "No file found".

The second one logs... "ci_scripts".

So something isn't taking that folder over to the test environment.

[oliverfoggin]

Oh man! I got there! @cameroncooke thanks so much for your help and comments. They got me there eventually.

So...

1. I discovered that ci_scripts does not necessarily go in the root of the repo. It lives as a sibling of the project file. So in my case it was in <repo_root>/App/ci_scripts.

2. I then realised that because of this my tests had to point to two separate directories.

• Locally it had to point to <repo_root>/App/ci_scripts/...
• In Xcode Cloud it points to <repo_root>/ci_scripts/...

Once that was done it all worked!

Got there eventually.

[oliverfoggin]

@lucasecf I don't know if this might be helpful to you too?

[lucasecf]

Didn't have exactly the same problem, but after debugging more I also found I had a wrong path for the snapshots on CI, so fixing that did the job for me 👌

[cameroncooke]

Wow that’s subtle. I’m glad you got it sorted in the end.

On Wed, 5 Oct 2022 at 22:15, Oliver Foggin ***@***.***> wrote: Oh man! I got there! @cameroncooke <https://github.com/cameroncooke> thanks so much for your help and comments. They got me there eventually. So... 1. I discovered that ci_scripts does not necessarily go in the root of the repo. It lives as a sibling of the project file. So in my case it was in <repo_root>/App/ci_scripts. 2. I then realised that because of this my tests had to point to two separate directories. - Locally it had to point to <repo_root>/App/ci_scripts/... - In Xcode Cloud it points to <repo_root>/ci_scripts/... Once that was done it all worked! Got there eventually. — Reply to this email directly, view it on GitHub <#553 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SNP3MCAWEFPAHQI7JTWBXVX5ANCNFSM5KSOPA2A> . You are receiving this because you were mentioned.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/3809123@ github.com>

-- Cameron.

[cameroncooke]

Um that looks okay, are you able to provide a demo project and I’ll take a look?

On Wed, 5 Oct 2022 at 19:58, Oliver Foggin ***@***.***> wrote: I’m not sure that is correct, when it says recording new snapshots and gives a path to the new snapshot that directory of the new snapshot is the location it was searching for and couldn’t fix the reference snapshot. The location of the new snapshot is... /Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests/testAnalytics.1.txt The snapshot directory I am giving it is... /Volumes/workspace/repository/ci_scripts/resources/Tests/AppFeatureTests/__Snapshots__/AppCoreTests — Reply to this email directly, view it on GitHub <#553 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAEZ6SK2UVNMUM27O2O7I3LWBXFTRANCNFSM5KSOPA2A> . You are receiving this because you were mentioned.Message ID: <pointfreeco/swift-snapshot-testing/repo-discussions/553/comments/3808223@ github.com>

-- Cameron.

[adamwulf]

I was having this same problem, and unfortunately didn't find this thread until after I'd spend a whole day finding a different solution. I opted to fork and change AssertSnapshot.swift and including the snapshot files in the test target, so that they're compiled into the test bundle.

Then in the AssertSnapshot.swift i change:

let snapshotFileUrl = snapshotDirectoryUrl
  .appendingPathComponent("\(testName).\(identifier)")
  .appendingPathExtension(snapshotting.pathExtension ?? "")

to:

let myBundle = Bundle(for: CleanCounterBetweenTestCases.self)
let snapshotFileUrl = myBundle.path(forResource: "\(testName).\(identifier)", ofType: snapshotting.pathExtension).map({ URL(fileURLWithPath: $0) })
  ?? snapshotDirectoryUrl
      .appendingPathComponent("\(testName).\(identifier)")
      .appendingPathExtension(snapshotting.pathExtension ?? "")

this has the benefit that the Test.swift files don't need to toggle the #file depending on CI.

[nikitaame]

@adamwulf how did you include the snapshot files in the test target?

[adamwulf]

In xcode, i select the image file(s), and in the File Inspector I make sure the test target is checked.

[rcarver]

Hello everyone, thanks for all the digging and research in this thread. I've created PR #716 that attempts to solve this in the library, hopefully in line with the patterns you're already using. Check it out and see if it would replace any ad-hoc solutions you're currently using?

[qmoya]

Hi there! We’ve been using this solution for over a year —see my old comment above—; since I notice people are still struggling, I’m open-sourcing a package that provides you with a drop-in replacement for assertSnapshot. I hope it’s helpful!

[adku]

Hello!
I'm struggling with making it work.

So far I tried everything, locally is working, but on Xcode Cloud I'm getting error:
No reference was found on disk. Automatically recorded snapshot: … open "/Volumes/workspace/repository/myProjectName/ci_scripts/Artifacts/__Snapshots__/MyTest/spec.myTest-screenshot.png" Re-run "spec" to test against the newly-recorded snapshot.

Right now symlink to ci_scripts root directory is configured. Snapshots is a symlink, real path myProjectName/Tests/.
Local + CI is using ci_scripts path based on this repo: https://github.com/invia-flights/swift-xcode-cloud-snapshot-testing

Generally it seems like similar situation to this post: https://developer.apple.com/forums/thread/716993, I'm running out of any new ideas.

Any suggestions what's more I can do in case to make it work on Xcode Cloud? Maybe something specific to Xcode Cloud workflow?

[adku]

Finally I figured out it!

There are two different locations.

One for Xcodebuild:
/Volumes/workspace/repository/YOUR_PROJECT/ci_scripts

Second one for Run tests on iPhone 11 (iOS 16.1) simulator:
/Volumes/workspace/repository/ci_scripts

As you can see for running tests location is root directory.

So for snapshotDirectory locations needs to be:
"/Volumes/workspace/repository/ci_scripts/Artifacts/Snapshots"
AND
Symlink needs to be created for ci_scripts in a root repo directory.

Here is repository with an inspiration and root ci_script symlink.
https://github.com/arvcpl/mycards/tree/v0.1.3-alpha

[jaanus]

I didn’t like the solutions around hacking ci_scripts. I got it to work by adding the screenshots to the test bundle instead. Wrote up my solution here: https://jaanus.com/snapshot-testing-xcode-cloud/

[DavidBrunow]

I like the solutions around hacking ci_scripts but I couldn't get things to work from this thread alone. I wrote a blog post with the steps that worked for me: https://davidbrunow.github.io/brunow.org/documentation/brunow/08-30-snapshot-testing-with-xcode-cloud

[rr-alvin-lee]

If anyone plans to use the ci_scripts hack, I would recommend this guide as it's pretty clear and easy to follow.

[fespinoza]

I made a guide on how to make this work https://youtu.be/KPJL8vN3lAg