Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Address test hanging in coverage workflow #17565

Closed
stinodego opened this issue Jul 11, 2024 · 0 comments · Fixed by #17566
Closed

Address test hanging in coverage workflow #17565

stinodego opened this issue Jul 11, 2024 · 0 comments · Fixed by #17566
Assignees
Labels
accepted Ready for implementation ci Related to the continuous integration setup P-medium Priority: medium

Comments

@stinodego
Copy link
Member

stinodego commented Jul 11, 2024

The Python coverage workflow regularly hangs - leading to the job taking 6 hours and then terminating due to the job time limit. We should figure out what causes this.

It happens when ASYNC is enabled. Here's some example logs:

ss.s.s.s................................................................ [  8%]
........................................................................ [ 16%]
........................................................................ [ 25%]
...............................................................s........ [ 33%]
..s..................................................................... [ 42%]
...................................s..s................................. [ 50%]
........................................................................ [ 58%]
thread 'polars-3' panicked at /Users/runner/work/polars/polars/crates/polars-io/src/parquet/read/read_impl.rs:763:58:
called `Result::unwrap()` on an `Err` value: (Ok([shape: (3, 2)
┌─────┬─────┐
│ a   ┆ b   │
│ --- ┆ --- │
│ i64 ┆ i64 │
╞═════╪═════╡
│ 1   ┆ 4   │
│ 2   ┆ 5   │
│ 3   ┆ 6   │
└─────┴─────┘]), 3, 18446744073709551612)
stack backtrace:
......................................................................... [ 67%]
......................................................sss.......xs...... [ 75%]
Fatal Python error: Aborted

Thread 0x0000700002968000 (most recent call first):
..[gw3] node down: Not properly terminated
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 534 in read
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 567 in from_io
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 1160 in _thread_receiver
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 341 in run
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn

Thread 0x00007ff859e9c7c0 (most recent call first):
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pyarrow/dataset.py", line 486 in _filesystem_dataset
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pyarrow/dataset.py", line 794 in dataset
  File "/Users/runner/work/polars/polars/py-polars/tests/unit/io/test_pyarrow_dataset.py", line 22 in helper_dataset_test
  File "/Users/runner/work/polars/polars/py-polars/tests/unit/io/test_pyarrow_dataset.py", line 150 in test_dataset_foo
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/python.py", line 162 in pytest_pyfunc_call
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/python.py", line 16[32](https://github.com/pola-rs/polars/actions/runs/9763018518/job/26947877229?pr=17359#step:12:33) in runtest
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 173 in pytest_runtest_call
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 241 in <lambda>
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 341 in from_call
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 240 in call_and_report
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 135 in runtestprotocol
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/runner.py", line 116 in pytest_runtest_protocol
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/xdist/remote.py", line 195 in run_one_test
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/xdist/remote.py", line 174 in pytest_runtestloop
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/main.py", line [33](https://github.com/pola-rs/polars/actions/runs/9763018518/job/26947877229?pr=17359#step:12:34)9 in _main
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/main.py", line 285 in wrap_session
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/_pytest/main.py", line 332 in pytest_cmdline_main
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_callers.py", line 103 in _multicall
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_manager.py", line 120 in _hookexec
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/pluggy/_hooks.py", line 513 in __call__
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/xdist/remote.py", line 393 in <module>
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 1291 in executetask
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line [34](https://github.com/pola-rs/polars/actions/runs/9763018518/job/26947877229?pr=17359#step:12:35)1 in run
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 411 in _perform_spawn
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 389 in integrate_as_primary_thread
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 1273 in serve
  File "/Users/runner/work/polars/polars/.venv/lib/python3.12/site-packages/execnet/gateway_base.py", line 1806 in serve
  File "<string>", line 8 in <module>
  File "<string>", line 1 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt199[37](https://github.com/pola-rs/polars/actions/runs/9763018518/job/26947877229?pr=17359#step:12:38), numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, yaml._yaml, markupsafe._speedups, sqlalchemy.cyextension.collections, sqlalchemy.cyextension.immutabledict, sqlalchemy.cyextension.processors, sqlalchemy.cyextension.resultproxy, sqlalchemy.cyextension.util, greenlet._greenlet, pyarrow.lib, zstandard.backend_c, pyarrow._fs, pyarrow._azurefs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._compute, pyarrow._acero, pyarrow._csv, pyarrow._json, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.hashing, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.internals, pandas._libs.indexing, pandas._libs.index, pandas._libs.writers, pandas._libs.join, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing, pyarrow._dataset, pyarrow._dataset_orc, pyarrow._parquet, pyarrow._parquet_encryption, pyarrow._dataset_parquet_encryption, pyarrow._dataset_parquet, adbc_driver_manager._lib, adbc_driver_manager._reader (total: 80)
F
replacing crashed worker gw3
..............s......................................................s.. [ 84%]
........................................................................ [ 92%]

Links to some failed workflows:

@stinodego stinodego added ci Related to the continuous integration setup P-medium Priority: medium labels Jul 11, 2024
@c-peters c-peters added the accepted Ready for implementation label Jul 15, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
accepted Ready for implementation ci Related to the continuous integration setup P-medium Priority: medium
Projects
Archived in project
Development

Successfully merging a pull request may close this issue.

3 participants