Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

The centralized nightlies job failed on Tuesday (2024-07-09) #14

Closed
github-actions bot opened this issue Jul 3, 2024 · 46 comments
Closed

The centralized nightlies job failed on Tuesday (2024-07-09) #14

github-actions bot opened this issue Jul 3, 2024 · 46 comments
Assignees
@jdblischak
Labels
nightly-failure The scheduled nightly builds failed

Comments

@github-actions
Copy link

github-actions bot commented Jul 3, 2024

The centralized nightlies job failed on Tuesday (2024-07-02) in run 9770774146
@github-actions github-actions bot added the nightly-failure The scheduled nightly builds failed label Jul 3, 2024
@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Tuesday (2024-07-09) in run 9867069920

@jdblischak
Copy link
Collaborator

Oops. I forgot to close this Issue yesterday. The original failure from last week was a spurious error when uploading an artifact. It has passed ever since then.

@jdblischak
Copy link
Collaborator

Last night's failure was due to the recent update to the dump functions

xref: TileDB-Inc/TileDB#5026, TileDB-Inc/conda-forge-nightly-controller#108, TileDB-Inc/TileDB-R#727, TileDB-Inc/TileDB-Py#1975

@jdblischak jdblischak changed the title The centralized nightlies job failed on Tuesday (2024-07-02) The centralized nightlies job failed on Tuesday (2024-07-09) Jul 10, 2024
@jdblischak jdblischak mentioned this issue Jul 10, 2024
Closed
@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Wednesday (2024-07-10) in run 9884378480

@jdblischak
Copy link
Collaborator

Still failing due to the dump error

@jdblischak
Copy link
Collaborator

Fix pending in TileDB-Inc/TileDB#5179

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Thursday (2024-07-11) in run 9901331468

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Friday (2024-07-12) in run 9916513178

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Saturday (2024-07-13) in run 9924615480

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Sunday (2024-07-14) in run 9932722986

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Monday (2024-07-15) in run 9949626990

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Tuesday (2024-07-16) in run 9967040160

@jdblischak
Copy link
Collaborator

Last night libtiledb failed due to a spurious download failure. Restarted

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Wednesday (2024-07-17) in run 9967040160

@jdblischak
Copy link
Collaborator

While the libtiledb failure last night was spurious, most of the client builds are now failing.

image

I suspect it is related to the recent removal of deprecated APIs in TileDB-Inc/TileDB#5146

I'm seeing similar error messages to those in other nightly builds

@jdblischak
Copy link
Collaborator

xref: TileDB-Inc/TileDB-VCF#747 (comment), TileDB-Inc/TileDB-R#730, TileDB-Inc/conda-forge-nightly-controller#108 (comment), TileDB-Inc/TileDB-MariaDB#358

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Wednesday (2024-07-17) in run 9984394452

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Thursday (2024-07-18) in run 10001437480

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Friday (2024-07-19) in run 10016749877

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Saturday (2024-07-20) in run 10024869559

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Sunday (2024-07-21) in run 10033500413

@jdblischak
Copy link
Collaborator

TileDB-Py and libtiledbvcf continue to fail due to the use of deprecated APIs. The tests for tiledbsoma-r sometimes pass and sometimes segfault. Unclear if this is spurious or not. Will have to keep monitoring

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Monday (2024-07-22) in run 10051518578

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Tuesday (2024-07-23) in run 10069345493

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Wednesday (2024-07-24) in run 10087006106

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Thursday (2024-07-25) in run 10104559745

@jdblischak
Copy link
Collaborator

The TileDB-Py error about tiledb_array_delete_fragments is gone (as expected since TileDB-Inc/TileDB-Py#2009 was merged).

However, there are still errors that are failing the build:

[12/33] Building CXX object tiledb/cc/CMakeFiles/cc.dir/array.cc.o
FAILED: tiledb/cc/CMakeFiles/cc.dir/array.cc.o
/usr/bin/g++ -Dcc_EXPORTS -isystem /opt/hostedtoolcache/Python/3.11.9/x64/include/python3.11 -isystem /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include -isystem /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/include -O3 -DNDEBUG -std=gnu++20 -fPIC -fvisibility=hidden -flto -fno-fat-lto-objects -MD -MT tiledb/cc/CMakeFiles/cc.dir/array.cc.o -MF tiledb/cc/CMakeFiles/cc.dir/array.cc.o.d -o tiledb/cc/CMakeFiles/cc.dir/array.cc.o -c /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc
/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc: In function ‘void libtiledbcpp::init_array(pybind11::module&)’:
/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc:100:76: error: no match for call to ‘(const pybind11::detail::overload_cast_impl<const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&, const tiledb::ArraySchema&, tiledb_encryption_type_t, const std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> >&>) (void (*)(const string&, const tiledb::ArraySchema&))’
   99 |            py::overload_cast<const std::string &, const ArraySchema &,
      |            ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  100 |                              tiledb_encryption_type_t, const std::string &>(
      |                              ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
  101 |                &Array::create))
      |                ~~~~~~~~~~~~~~~
In file included from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/attr.h:13,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/detail/class.h:12,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/pybind11.h:13,
                 from /tmp/pip-build-env-gsspe4gg/overlay/lib/python3.11/site-packages/pybind11/include/pybind11/numpy.h:12,
                 from /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/common.h:4,
                 from /home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/tiledb/cc/array.cc:4:

The error appears related to pybind11. @teo-tsirpanis do you have any ideas what could be causing this?

@teo-tsirpanis
Copy link
Member

Apparently one API was missed in TileDB-Inc/TileDB-Py#1958, c.c. @kounelisagis.

@kounelisagis kounelisagis mentioned this issue Jul 26, 2024
Merged
@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Friday (2024-07-26) in run 10120050775

@teo-tsirpanis
Copy link
Member

Oh my goodness, another one? 😩

@jdblischak
Copy link
Collaborator

Do we have to remove all instances of tiledb_encryption_type_t? If yes, I see 3 more instances in tiledb/cc/array.cc that remain after TileDB-Inc/TileDB-Py#2021 was merged:

https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L124
https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L40
https://github.com/TileDB-Inc/TileDB-Py/blob/e29bf7301c44df9c8086c83a66deaf3c9089e12d/tiledb/cc/array.cc#L45

cc: @kounelisagis

@jdblischak jdblischak mentioned this issue Jul 27, 2024
Closed
@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Saturday (2024-07-27) in run 10128189890

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Sunday (2024-07-28) in run 10136948012

@kounelisagis kounelisagis mentioned this issue Jul 29, 2024
Merged
@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Monday (2024-07-29) in run 10154979104

@github-actions GitHub Actions
Copy link
Author

The centralized nightlies job failed on Tuesday (2024-07-30) in run 10172798362

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 1, 2024

The centralized nightlies job failed on Wednesday (2024-07-31) in run 10190812724

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 2, 2024

The centralized nightlies job failed on Thursday (2024-08-01) in run 10208863772

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 3, 2024

The centralized nightlies job failed on Friday (2024-08-02) in run 10224480159

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 4, 2024

The centralized nightlies job failed on Saturday (2024-08-03) in run 10232732546

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 5, 2024

The centralized nightlies job failed on Sunday (2024-08-04) in run 10241445902

@kounelisagis
Copy link
Member

For Python, should we do something similar to this TileDB-Inc/conda-forge-nightly-controller#122 @jdblischak?

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 6, 2024

The centralized nightlies job failed on Monday (2024-08-05) in run 10259108970

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 7, 2024

The centralized nightlies job failed on Tuesday (2024-08-06) in run 10276973841

@github-actions GitHub Actions
Copy link
Author

github-actions bot commented Aug 8, 2024

The centralized nightlies job failed on Wednesday (2024-08-07) in run 10294780430

@jdblischak
Copy link
Collaborator

It's been a bit of a roller coaster ride, but things are settling down.

3 days ago there were 4 failings builds:

image

But last night only TileDB-Py failed. As @kounelisagis suggested above (https://github.com/jdblischak/centralized-tiledb-nightlies/issues/14#issuecomment-2268942904), I'll try addressing this with -Cskbuild.cmake.define.TILEDB_REMOVE_DEPRECATIONS=OFF.

The libtiledbvcf build was fixed by TileDB-Inc/TileDB-VCF#753 and TileDB-Inc/TileDB-VCF#755

Unclear why TileDB-Go is passing now. 3 days ago it was clearly failing due to using deprecated APIs (see test failure below). However, while there have been recent commits both to TileDB-Go and TileDB, none appear related to these particular deprecated APIs.

# github.com/TileDB-Inc/TileDB-Go [github.com/TileDB-Inc/TileDB-Go.test]
cgo-gcc-prolog: In function ‘_cgo_b6efb34e80b0_Cfunc_tiledb_enumeration_dump’:
cgo-gcc-prolog:252:2: warning: ‘tiledb_enumeration_dump’ is deprecated [-Wdeprecated-declarations]
In file included from ./install-libtiledb/include/tiledb/tiledb_experimental.h:46,
                 from ./enumeration_experimental.go:5:
./install-libtiledb/include/tiledb/api/c_api/enumeration/enumeration_api_experimental.h:314:40: note: declared here
  314 | TILEDB_DEPRECATED_EXPORT capi_return_t tiledb_enumeration_dump(
      |                                        ^~~~~~~~~~~~~~~~~~~~~~~

Also unclear why TileDB-R is passing now. Maybe the previous error was spurious. There have been no recent upstream commits.

g++ -std=gnu++17 -shared -L/usr/lib/R/lib -Wl,-Bsymbolic-functions -flto=auto -ffat-lto-objects -flto=auto -Wl,-z,relro -o tiledb.so RcppExports.o arrowio.o batched.o column_buffer.o deprecation.o durations.o libtiledb.o nanoarrow.o nullable.o shmem.o virtualfile.o -L/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib -ltiledb -Wl,-rpath,/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib -L/usr/lib/R/lib -lR
/usr/bin/ld:/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib/libtiledb.so: file format not recognized; treating as linker script
/usr/bin/ld:/home/runner/work/centralized-tiledb-nightlies/centralized-tiledb-nightlies/install-libtiledb/lib/libtiledb.so:0: syntax error
collect2: error: ld returned 1 exit status
make: *** [/usr/share/R/share/make/shlib.mk:10: tiledb.so] Error 1
ERROR: compilation failed for package ‘tiledb’
* removing ‘/usr/local/lib/R/site-library/tiledb’

@jdblischak
Copy link
Collaborator

TileDB-Py build fixed by ec5ce1f (build). Thanks @kounelisagis!

xref: TileDB-Inc/TileDB-Py#2023, TileDB-Inc/conda-forge-nightly-controller#122

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jdblischak jdblischak

Labels
nightly-failure The scheduled nightly builds failed
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@jdblischak @teo-tsirpanis @kounelisagis