Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[CI][Python][C++] Substrait integration tests are crashing since substrait-python 0.4.1 release (substrait spec v0.34.0) #37932

Closed
jorisvandenbossche opened this issue Sep 28, 2023 · 2 comments
Closed

[CI][Python][C++] Substrait integration tests are crashing since substrait-python 0.4.1 release (substrait spec v0.34.0) #37932

jorisvandenbossche opened this issue Sep 28, 2023 · 2 comments
Labels
Component: Continuous Integration Component: Python

Comments

@jorisvandenbossche
Copy link
Member

The substrait integration tests started to fail last night (https://github.com/ursacomputing/crossbow/actions/runs/6332972733/job/17200364355) with a segmentation fault (the logs don't give much detail beyond that):

 ============================= test session starts ==============================
platform linux -- Python 3.10.12, pytest-7.4.2, pluggy-1.3.0
rootdir: /consumer-testing
plugins: snapshot-0.9.0, xdist-3.3.1, cython-0.2.1, hypothesis-6.86.2, lazy-fixture-0.6.3
collected 60 items

substrait_consumer/tests/functional/extension_functions/test_boolean_functions.py s [  1%]
Fatal Python error: Segmentation fault

Current thread 0x00007ff957150740 (most recent call first):
  File "/consumer-testing/substrait_consumer/context.py", line 26 in produce_isthmus_substrait
  File "/consumer-testing/substrait_consumer/producers.py", line 122 in produce_substrait
  File "/consumer-testing/substrait_consumer/functional/common.py", line 137 in substrait_producer_function_test
  File "/consumer-testing/substrait_consumer/tests/functional/extension_functions/test_boolean_functions.py", line 54 in test_producer_boolean_functions
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/opt/conda/envs/arrow/bin/pytest", line 10 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._hdfsio, pyarrow._parquet, pyarrow._fs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._compute, pyarrow._substrait, google._upb._message, yaml._yaml, _jpype, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing (total: 64)
Fatal Python error: Aborted

Current thread 0x00007ff957150740 (most recent call first):
  File "/consumer-testing/substrait_consumer/context.py", line 26 in produce_isthmus_substrait
  File "/consumer-testing/substrait_consumer/producers.py", line 122 in produce_substrait
  File "/consumer-testing/substrait_consumer/functional/common.py", line 137 in substrait_producer_function_test
  File "/consumer-testing/substrait_consumer/tests/functional/extension_functions/test_boolean_functions.py", line 54 in test_producer_boolean_functions
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/python.py", line 194 in pytest_pyfunc_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/python.py", line 1792 in runtest
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 169 in pytest_runtest_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 262 in <lambda>
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 341 in from_call
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 261 in call_runtest_hook
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 222 in call_and_report
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 133 in runtestprotocol
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/runner.py", line 114 in pytest_runtest_protocol
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 350 in pytest_runtestloop
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 325 in _main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 271 in wrap_session
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/main.py", line 318 in pytest_cmdline_main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_callers.py", line 77 in _multicall
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_manager.py", line 115 in _hookexec
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/pluggy/_hooks.py", line 493 in __call__
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/config/__init__.py", line 169 in main
  File "/opt/conda/envs/arrow/lib/python3.10/site-packages/_pytest/config/__init__.py", line 192 in console_main
  File "/opt/conda/envs/arrow/bin/pytest", line 10 in <module>

Extension modules: numpy.core._multiarray_umath, numpy.core._multiarray_tests, numpy.linalg._umath_linalg, numpy.fft._pocketfft_internal, numpy.random._common, numpy.random.bit_generator, numpy.random._bounded_integers, numpy.random._mt19937, numpy.random.mtrand, numpy.random._philox, numpy.random._pcg64, numpy.random._sfc64, numpy.random._generator, pyarrow.lib, pyarrow._hdfsio, pyarrow._parquet, pyarrow._fs, pyarrow._hdfs, pyarrow._gcsfs, pyarrow._s3fs, pyarrow._compute, pyarrow._substrait, google._upb._message, yaml._yaml, _jpype, pandas._libs.tslibs.np_datetime, pandas._libs.tslibs.dtypes, pandas._libs.tslibs.base, pandas._libs.tslibs.nattype, pandas._libs.tslibs.timezones, pandas._libs.tslibs.ccalendar, pandas._libs.tslibs.fields, pandas._libs.tslibs.timedeltas, pandas._libs.tslibs.tzconversion, pandas._libs.tslibs.timestamps, pandas._libs.properties, pandas._libs.tslibs.offsets, pandas._libs.tslibs.strptime, pandas._libs.tslibs.parsing, pandas._libs.tslibs.conversion, pandas._libs.tslibs.period, pandas._libs.tslibs.vectorized, pandas._libs.ops_dispatch, pandas._libs.missing, pandas._libs.hashtable, pandas._libs.algos, pandas._libs.interval, pandas._libs.lib, pandas._libs.ops, pandas._libs.arrays, pandas._libs.tslib, pandas._libs.sparse, pandas._libs.indexing, pandas._libs.index, pandas._libs.internals, pandas._libs.join, pandas._libs.writers, pandas._libs.window.aggregations, pandas._libs.window.indexers, pandas._libs.reshape, pandas._libs.groupby, pandas._libs.json, pandas._libs.parsers, pandas._libs.testing (total: 64)
/arrow/ci/scripts/integration_substrait.sh: line 31:  9641 Aborted                 (core dumped) pytest -r s substrait_consumer/tests/functional/extension_functions/test_boolean_functions.py --producer IsthmusProducer --consumer AceroConsumer
134
Error: `docker-compose --file /home/runner/work/crossbow/crossbow/arrow/docker-compose.yml run --rm -e SETUPTOOLS_SCM_PRETEND_VERSION=14.0.0.dev201 conda-python-substrait` exited with a non-zero exit code 134, see the process log above.

...

Archery was called with:
  export PYTHON=3.10
sssssssssssssssssssssssssssssssssssssssss...sss
Error: Process completed with exit code 1.

Yesterday the build was still passing (https://github.com/ursacomputing/crossbow/actions/runs/6319792351/job/17161308535), and one obvious difference is that the failing build started to use substrait-python 0.4.1 that was just released (https://github.com/substrait-io/substrait-python/releases/tag/v0.4.1) and which updated substrait to v0.34.0
@jorisvandenbossche jorisvandenbossche mentioned this issue Sep 28, 2023
Closed
@jorisvandenbossche
Copy link
Member Author

jorisvandenbossche commented Sep 29, 2023 •
edited
Loading

From the discussion in substrait-io/substrait-python#25, we might want to pin the version of substrait python package so the substrait version matches with what we have pinned in the Arrow C++ build:

ARROW_SUBSTRAIT_BUILD_VERSION=v0.27.0

Or at least to a version of substrait-python which included a substrait spec version that is known not to clash, as it seems substrait-python doesn't go that far back as substrait v0.27.0

jorisvandenbossche added a commit to jorisvandenbossche/arrow that referenced this issue Sep 29, 2023
@jorisvandenbossche
apacheGH-37932 [CI] Pin substrait python package to fix nightly subst…
b8d21fc 
…rait integration build
@jorisvandenbossche
Copy link
Member Author

It seems this "resolved" itself ..

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
Component: Continuous Integration Component: Python
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@jorisvandenbossche