Error: Command failed: df -Pk when using macos-14 runners (macos-latest)

[ebickle]

After GitHub migrated the macos-latest label to MacOS 14 (from 12) we noticed the following warning in our CodeQL logs:

arning: Failed to check available disk space: Error: Command failed: df -Pk -- /Users/runner/work/repo-name/directory-name
dyld[4412]: terminating because inserted dylib '/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' could not be loaded: tried: '/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e'))
dyld[4412]: tried: '/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/hostedtoolcache/CodeQL/2.17.6/arm64/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need 'arm64e'))

There isn't too much to the workflow up until that point - it runs actions/checkout, pod install, then github/codeql-action/init with languages:swift on a runner: macos-latest.

As a double check, I cleared all of the GitHub workflow caches and verified I see "Did not find CodeQL tools version 2.17.6 in the toolcache" in the build output.

[ginsbach]

May I ask which version of CodeQL Action you are using? We have previously fixed a similar-looking issue, so it might be sufficient for you to upgrade to the newest version.

[ebickle]

Hi @ginsbach, the workflow in question is using v3 of init and analyze:
github/codeql-action/init@v3
github/codeql-action/analyze@v3

The build itself is a direct call to XCode:
xcodebuild build -workspace "app/app.xcworkspace" -scheme "${{ env.scheme }}" -destination "${{env.destination }}"

[angelapwen]

👋 thank you for the report @ebickle. The warning about df should be unrelated to the actual failure – the PR in the CodeQL Action that @ginsbach linked above is in fact where we made the df command failure a warning rather than a failure.

That said, the following lines look like a new issue with the build tracer. We've raised it with the appropriate team internally!

[criemen]

Hi @ebickle , any chance you could rerun your workflow in debug mode and provide the debug artifact to us?

If it contains sensitive data, reach out to github support to privately send it to us, otherwise attaching it on this issue is also fine.

[ebickle]

@criemen I've uploaded the raw debug logs over at https://support.github.com/ticket/personal/0/2886802

[criemen]

@ebickle Can you upload the debug artifact, not the debug logs? The codeql-action should produce a debug artifact. If that contains too much confidential data, I'm mostly interested in the build-tracer.log from <database>/log/build-tracer.log, but the whole thing would be useful.

[ebickle]

@criemen Artifacts attached now too!

[criemen]

Thanks, that was very helpful! We're discussing the problem and possible solutions internally and will get back to you.

[angelapwen]

Thanks to @criemen we have a better understanding of the problem related to the build tracer here. We'll work on a fix in the CodeQL Action and update here when we have one!

[angelapwen]

👋 we've just released v3 and v2 of the CodeQL Action with this change: github/codeql-action#2434, which removes our call to df and therefore should prevent these warnings 😸 @ebickle do you mind checking to see if this also resolves the warnings on your end?

[ebickle]

Still seeing issues on my side, but different warnings and errors. I've tossed the details into support ticket # 2886802. It occurs when running github/codeql-action/init@v3 on with the languages: swift input set on a macos-latest runner. The csrutil command fails with a similar error about an incompatible CPU architecture. The null exit code (due to the failure) causes the warning Warning: Failed to determine if System Integrity Protection was enabled

[angelapwen]

Got it — I see, the csrutil command is going to fail for the same reason. I'll make that change shortly and release. Thanks for the quick response!

[angelapwen]

github/codeql-action#2441 should have the fix in!

[akoeplinger]

@angelapwen we're also seeing this issue when running df ourselves whenever CodeQL is active and it fails our build, does that mean we need to stop running that command or are you also working on fixing the build tracer?

+ df -h
dyld[4958]: terminating because inserted dylib '/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' could not be loaded: tried: '/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need ''))
dyld[4958]: tried: '/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need '')), '/System/Volumes/Preboot/Cryptexes/OS/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (no such file), '/Users/runner/work/_temp/codeql3000/distribution/codeql/tools/osx64/libtrace.dylib' (fat file, but missing compatible architecture (have 'x86_64,arm64', need ''))
/Users/runner/work/_temp/7a3e0e0b-310b-4b2e-8281-5194cb8b809b.sh: line 2:  4958 Abort trap: 6           df -h

##[error]Bash exited with code '134'.

For context this is on an internal Azure DevOps project, feel free to ping me on Teams if you need details.

[angelapwen]

Thanks for checking in @akoeplinger — the build tracer fix would likely be much more complex and is not currently scheduled. I recommend stop running the command for now (on MacOS ARM machines) until the work is able to be prioritized.

I believe there are some established Microsoft/CodeQL lines of support/requests. Depending on how important running the command is for you, you could raise the issue there to be prioritized as well.

[ebickle]

github/codeql-action#2441 should have the fix in!

That did the trick! Tested against @main and everything looks clear now. Thanks!

[angelapwen]

Perfect, thank you for letting us know! I'll kick off a release to get it into v3 and v2.

[akoeplinger]

@angelapwen there is https://portal.microsofticm.com/imp/v5/incidents/details/532165079/summary but not sure if there are other open support tickets as that one is still in Triage status.

Also my understanding is that this will happen for every platform binary which is arm64e so it'd probably be good to have a public tracking issue.

I tried removing df from our build but it just hits some other binary, this isn't really workable.