Linker SEGV on MacOS, as of today #6350

johnkerl · 2022-10-03T23:12:18Z

Description

Happening on https://github.com/single-cell-data/TileDB-SOMA as of today
MacOS CI runs affected -- for example https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104
As a test I tried putting up a PR branched off an earlier commit which was green in CI at the time -- from there I only added a comment mod
- [ci] Debugging MacOS CI XCode 14 linker issues (please do not review) single-cell-data/TileDB-SOMA#368
- https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104
- Since this code was green in CI before, and red in CI only with a comment mod, I am certain this is not due to any change in our code

Linker stack trace from https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104:

  [100%] Linking CXX executable example

  -- stderr output is:
  0  0x10259dffa  __assert_rtn + 139
  1  0x1023d128d  mach_o::relocatable::Parser<x86_64>::parse(mach_o::relocatable::ParserOptions const&) + 4989
  2  0x1023c1f8f  mach_o::relocatable::Parser<x86_64>::parse(unsigned char const*, unsigned long long, char const*, long, ld::File::Ordinal, mach_o::relocatable::ParserOptions const&) + 207
  3  0x1024389d4  ld::tool::InputFiles::makeFile(Options::FileInfo const&, bool) + 2036
  4  0x10243bfa0  ___ZN2ld4tool10InputFilesC2ER7Options_block_invoke + 48
  5  0x7ff814b4e34a  _dispatch_client_callout2 + 8
  6  0x7ff814b60c45  _dispatch_apply_invoke_and_wait + 213
  7  0x7ff814b60161  _dispatch_apply_with_attr_f + 1178
  8  0x7ff814b60327  dispatch_apply + 45
  9  0x10243be2d  ld::tool::InputFiles::InputFiles(Options&) + 669
  10  0x1023acd48  main + 840
  A linker snapshot was created at:
        /tmp/example-2022-10-03-224834.ld-snapshot
  ld: Assertion failed: (_file->_atomsArrayCount == computedAtomCount && "more atoms allocated than expected"), function parse, file macho_relocatable_file.cpp, line 2061.
  collect2: error: ld returned 1 exit status
  make[5]: *** [example/example] Error 1
  make[4]: *** [example/CMakeFiles/example.dir/all] Error 2
  make[3]: *** [all] Error 2

  CMake Error at /Users/runner/work/TileDB-SOMA/TileDB-SOMA/build/externals/src/ep_spdlog-stamp/ep_spdlog-build-Release.cmake:47 (message):
    Stopping after outputting logs.

This appears to be an assertion failure inside the linker itself.

Platforms affected

Azure DevOps
GitHub Actions

Runner images affected

Image version and build link

https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104

Current runner version: '2.297.0'
Operating System
Runner Image
  Image: macos-[1](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104#step:1:1)[2](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104#step:1:2)
  Version: 20220929.2
  Included Software: https://github.com/actions/runner-images/blob/macOS-12/20220[9](https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3177996393/jobs/5179048104#step:1:10)29.2/images/macos/macos-12-Readme.md
  Image Release: https://github.com/actions/runner-images/releases/tag/macOS-12%2F20220929.2

Is it regression?

https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3174959251/jobs/5172405039

Expected behavior

The linker shouldn't fail an assertion while compiling C++ code

Actual behavior

Assertion-fail in the linker

Repro steps

Publish [ci] Debugging MacOS CI XCode 14 linker issues (please do not review) single-cell-data/TileDB-SOMA#368
Look at CI fail for MacOS

The text was updated successfully, but these errors were encountered:

johnkerl · 2022-10-03T23:49:22Z

Sometimes when we 're-run failed jobs' we get the same linker-assertion-failure message; other times we do not. I can't find a pattern other than to say there appears to be some randomness involved ...

johnkerl · 2022-10-04T03:10:07Z

As a workaround we've downgraded our CI runner from macos-12 to macos-11

single-cell-data/TileDB-SOMA#370

johnkerl · 2022-10-04T03:19:23Z

Ah, more info: successful build log

2022-10-04T01:09:58.4829685Z Requested labels: macos-12
2022-10-04T01:09:58.4829732Z Job defined at: single-cell-data/TileDB-SOMA/.github/workflows/cpp-ci.yml@refs/pull/369/merge
2022-10-04T01:09:58.4829754Z Waiting for a runner to pick up this job...
2022-10-04T01:09:59.5552729Z Job is waiting for a hosted runner to come online.
2022-10-04T01:10:07.4862683Z Job is about to start running on the hosted runner: GitHub Actions 5 (hosted)
2022-10-04T01:10:11.8069710Z Current runner version: '2.297.0'
2022-10-04T01:10:11.8111420Z ##[group]Operating System
2022-10-04T01:10:11.8112110Z macOS
2022-10-04T01:10:11.8112330Z 12.6
2022-10-04T01:10:11.8112520Z 21G115
2022-10-04T01:10:11.8112740Z ##[endgroup]
2022-10-04T01:10:11.8112980Z ##[group]Runner Image
2022-10-04T01:10:11.8113240Z Image: macos-12
2022-10-04T01:10:11.8113470Z Version: 20220917.1
...

and unsuccessful build log

2022-10-03T22:45:11.8150285Z Requested labels: macos-12
2022-10-03T22:45:11.8150330Z Job defined at: single-cell-data/TileDB-SOMA/.github/workflows/cpp-ci.yml@refs/pull/368/merge
2022-10-03T22:45:11.8150778Z Waiting for a runner to pick up this job...
2022-10-03T22:45:12.5907201Z Job is waiting for a hosted runner to come online.
2022-10-03T22:45:21.6927097Z Job is about to start running on the hosted runner: GitHub Actions 7 (hosted)
2022-10-03T22:45:27.0600950Z Current runner version: '2.297.0'
2022-10-03T22:45:27.0655070Z ##[group]Operating System
2022-10-03T22:45:27.0655910Z macOS
2022-10-03T22:45:27.0656260Z 12.6
2022-10-03T22:45:27.0656590Z 21G115
2022-10-03T22:45:27.0656980Z ##[endgroup]
2022-10-03T22:45:27.0657370Z ##[group]Runner Image
2022-10-03T22:45:27.0657810Z Image: macos-12
2022-10-03T22:45:27.0658210Z Version: 20220929.2

mikhailkoliada · 2022-10-04T07:15:27Z

@johnkerl hello! We will take a look!

First idea is a bug in xcode, have you tried to choose a different xcode version in your pipeline?

burnpanck · 2022-10-04T09:28:56Z

I observe the same assertion in ld trying to compile something completely different locally on my mac (on aarch64 with macOS 12.6 / xcode 14). The same code compiled previously. In my case, the assert only happens when I compile in debug mode (I use GCC; the error happens both with GCC 11 and GCC 12). Given that it is an assert, it most definitely should be considered a bug (specifically in ld, which is comes from xcode, not from GCC). I have reported the error back to Apple about 10 days ago, so far I haven't received any feedback. Extremely annoying.

johnkerl · 2022-10-04T11:43:10Z

@johnkerl hello! We will take a look!

First idea is a bug in xcode, have you tried to choose a different xcode version in your pipeline?

@mikhailkoliada as a workaround we've downgraded our CI runner from macos-12 to macos-11

mikhailkoliada · 2022-10-04T11:54:09Z

@johnkerl could you please try to get back to macos-12 and try different xcode version(s)?

johnkerl · 2022-10-04T14:06:04Z

@mikhailkoliada hmm I tried
https://github.com/single-cell-data/TileDB-SOMA/pull/368/files
but still the fail at
https://github.com/single-cell-data/TileDB-SOMA/actions/runs/3182721862/jobs/5189036547
wherein I see

...
  CMake Error at /Users/runner/work/TileDB-SOMA/TileDB-SOMA/build/externals/src/ep_spdlog-stamp/ep_spdlog-build-Release.cmake:37 (message):
    Command failed: 2

     '/Applications/Xcode_14.0.1.app/Contents/Developer/usr/bin/make'
...
  ld: Assertion failed: (_file->_atomsArrayCount == computedAtomCount && "more atoms allocated than expected"), function parse, file macho_relocatable_file.cpp, line 2061.
...

so I'll need to check my understand of how to configure XCode versioning within GHA.

eseiler · 2022-10-05T19:10:41Z

We had a similar issue with our tests. Running sudo xcode-select -s '/Applications/Xcode_13.4.1.app/Contents/Developer' (the default in 20220917.1) resolved it.

dmitry-shibanov · 2022-10-06T12:17:44Z

Hello everyone. It looks like an issue with Xcode, that is why for now I'm going to close the issue.

al-cheb · 2022-10-06T12:22:54Z

We had a similar issue with our tests. Running sudo xcode-select -s '/Applications/Xcode_13.4.1.app/Contents/Developer' (the default in 20220917.1) resolved it.

Our recommendation is to use this actions - https://github.com/marketplace/actions/setup-xcode-version

Apparently, xcode 14 has a reported linker bug which prevents a successful build. The recommended workaround is to revert toolchain to last known good version (v13.4.1). More information can be found here - actions/runner-images#6350

Apparently, xcode 14 has a reported linker bug which prevents a successful build. The recommended workaround is to revert toolchain to last known good version (v13.4.1). More information can be found here - actions/runner-images#6350 Also, when macos-latest is used, CI can use either macos11 or macos12 and the build fails if it selects the former. Enforce macos-12 strictly for now.

johnkerl added bug report needs triage labels Oct 3, 2022

johnkerl mentioned this issue Oct 4, 2022

[ci] Work around recent Mac GHA issue single-cell-data/TileDB-SOMA#370

Merged

mikhailkoliada added Area: Apple Area: Common Tools investigate Collect additional information, like space on disk, other tool incompatibilities etc. and removed needs triage labels Oct 4, 2022

johnkerl mentioned this issue Oct 4, 2022

[ci] Debugging MacOS CI XCode 14 linker issues (please do not review) single-cell-data/TileDB-SOMA#368

Closed

johnkerl mentioned this issue Oct 4, 2022

[ci] Fix scripts/test for MacOS single-cell-data/TileDB-SOMA#358

Merged

dmitry-shibanov closed this as completed Oct 6, 2022

eseiler mentioned this issue Oct 13, 2022

Github Action macOS-12 runner Xcode build failing with no log #6378

Closed

11 tasks

johnkerl mentioned this issue Oct 15, 2022

[ci] Use MacOS 12 for CI single-cell-data/TileDB-SOMA#413

Merged

quentin mentioned this issue Oct 20, 2022

try to workaround xcode 14 issue souffle-lang/souffle#2331

Merged

hs-apotell mentioned this issue Oct 21, 2022

Fix mac builds ... once again chipsalliance/Surelog#3281

Merged

skyler-rs mentioned this issue Oct 24, 2022

update macos-gcc to gcc-11 os-fpga/FOEDAG#752

Merged

vineetbansal mentioned this issue Nov 17, 2022

Running CI on macos-11 + macos-latest osqp/osqp#490

Merged

ilia-shipitsin mentioned this issue May 8, 2023

macos-13 job crashes without logs #7509

Closed

9 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Linker SEGV on MacOS, as of today #6350

Linker SEGV on MacOS, as of today #6350

johnkerl commented Oct 3, 2022

johnkerl commented Oct 3, 2022 •

edited

Loading

johnkerl commented Oct 4, 2022

johnkerl commented Oct 4, 2022

mikhailkoliada commented Oct 4, 2022

burnpanck commented Oct 4, 2022 •

edited

Loading

johnkerl commented Oct 4, 2022

mikhailkoliada commented Oct 4, 2022

johnkerl commented Oct 4, 2022

eseiler commented Oct 5, 2022

dmitry-shibanov commented Oct 6, 2022

al-cheb commented Oct 6, 2022

Linker SEGV on MacOS, as of today #6350

Linker SEGV on MacOS, as of today #6350

Comments

johnkerl commented Oct 3, 2022

Description

Platforms affected

Runner images affected

Image version and build link

Is it regression?

Expected behavior

Actual behavior

Repro steps

johnkerl commented Oct 3, 2022 • edited Loading

johnkerl commented Oct 4, 2022

johnkerl commented Oct 4, 2022

mikhailkoliada commented Oct 4, 2022

burnpanck commented Oct 4, 2022 • edited Loading

johnkerl commented Oct 4, 2022

mikhailkoliada commented Oct 4, 2022

johnkerl commented Oct 4, 2022

eseiler commented Oct 5, 2022

dmitry-shibanov commented Oct 6, 2022

al-cheb commented Oct 6, 2022

johnkerl commented Oct 3, 2022 •

edited

Loading

burnpanck commented Oct 4, 2022 •

edited

Loading