Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GH-45006: [CI][Python] Fix test_memory failures #45007

Merged
merged 5 commits into from
Dec 13, 2024
Merged

GH-45006: [CI][Python] Fix test_memory failures #45007

merged 5 commits into from
Dec 13, 2024

Conversation

pitrou
Copy link
Member

@pitrou pitrou commented Dec 12, 2024 •
edited by github-actions bot
Loading

test_memory.py has started failing on some builds after #44951 was merged

@pitrou
Copy link
Member Author

pitrou commented Dec 12, 2024

@github-actions crossbow submit -g python

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@pitrou pitrou mentioned this pull request Dec 12, 2024
Merged
@pitrou
Copy link
Member Author

pitrou commented Dec 12, 2024

@github-actions crossbow submit -g python

@pitrou
Copy link
Member Author

pitrou commented Dec 12, 2024 •
edited
Loading

@github-actions crossbow submit wheelcp312*

@github-actions GitHub Actions
Copy link

Revision: 7dbaa32

Submitted crossbow builds: ursacomputing/crossbow @ actions-1325385d4d

Task Status
wheel-macos-monterey-cp312-cp312-amd64 GitHub Actions
wheel-macos-monterey-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2-28-cp312-cp312-arm64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-amd64 GitHub Actions
wheel-manylinux-2014-cp312-cp312-arm64 GitHub Actions
wheel-windows-cp312-amd64 GitHub Actions

@github-actions GitHub Actions

This comment was marked as outdated.

Sign in to view
@apache apache deleted a comment from github-actions bot Dec 12, 2024
@pitrou pitrou marked this pull request as ready for review December 12, 2024 12:22
raulcd
raulcd reviewed Dec 12, 2024
View reviewed changes
Copy link
Member

@raulcd raulcd left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The change LGTM, I am just wondering, is there any case where we test jemalloc for pyarrow? Do we want to?

@github-actions github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Dec 12, 2024
@pitrou
Copy link
Member Author

pitrou commented Dec 13, 2024

The change LGTM, I am just wondering, is there any case where we test jemalloc for pyarrow?

What do you mean? It's still tested in test_memory.py, it's just not mandatory anymore.

@jorisvandenbossche
Copy link
Member

You turned the default in ci/scripts/cpp_build.sh to OFF, so I think the question is whether we are then actually still testing this in any of the CI?

From a quick look, I see that the python and c++ macos github action build have a ARROW_JEMALLOC: ON env variable that would override this.

@pitrou
Copy link
Member Author

pitrou commented Dec 13, 2024

Ah, perhaps we should ensure that it's enabled on some CI tests indeed. Will take a look.

@pitrou pitrou force-pushed the gh44951-fix-memory-test branch from 7dbaa32 to 2ac2b8e Compare December 13, 2024 13:18
@github-actions github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Dec 13, 2024
@pitrou
Copy link
Member Author

pitrou commented Dec 13, 2024

@github-actions crossbow submit -g python -g cpp

@raulcd
Copy link
Member

raulcd commented Dec 13, 2024

Yes, sorry, that's what I meant. If we were enabling jemalloc for Python builds on any of our CI builds.

@github-actions GitHub Actions
Copy link

Revision: 2ac2b8e

Submitted crossbow builds: ursacomputing/crossbow @ actions-42c1c96121

Task Status
example-cpp-minimal-build-static GitHub Actions
example-cpp-minimal-build-static-system-dependency GitHub Actions
example-cpp-tutorial GitHub Actions
example-python-minimal-build-fedora-conda GitHub Actions
example-python-minimal-build-ubuntu-venv GitHub Actions
test-alpine-linux-cpp GitHub Actions
test-build-cpp-fuzz GitHub Actions
test-conda-cpp GitHub Actions
test-conda-cpp-valgrind GitHub Actions
test-conda-python-3.10 GitHub Actions
test-conda-python-3.10-cython2 GitHub Actions
test-conda-python-3.10-hdfs-2.9.2 GitHub Actions
test-conda-python-3.10-hdfs-3.2.1 GitHub Actions
test-conda-python-3.10-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.10-substrait GitHub Actions
test-conda-python-3.11 GitHub Actions
test-conda-python-3.11-dask-latest GitHub Actions
test-conda-python-3.11-dask-upstream_devel GitHub Actions
test-conda-python-3.11-hypothesis GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-1.26 GitHub Actions
test-conda-python-3.11-pandas-latest-numpy-latest GitHub Actions
test-conda-python-3.11-pandas-nightly-numpy-nightly GitHub Actions
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly GitHub Actions
test-conda-python-3.11-spark-master GitHub Actions
test-conda-python-3.12 GitHub Actions
test-conda-python-3.12-cpython-debug GitHub Actions
test-conda-python-3.13 GitHub Actions
test-conda-python-3.9 GitHub Actions
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5 GitHub Actions
test-conda-python-emscripten GitHub Actions
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2 GitHub Actions
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-cuda-python-ubuntu-22.04-cuda-11.7.1 GitHub Actions
test-debian-12-cpp-amd64 GitHub Actions
test-debian-12-cpp-i386 GitHub Actions
test-debian-12-python-3-amd64 GitHub Actions
test-debian-12-python-3-i386 GitHub Actions
test-fedora-39-cpp GitHub Actions
test-fedora-39-python-3 GitHub Actions
test-ubuntu-20.04-cpp GitHub Actions
test-ubuntu-20.04-cpp-bundled GitHub Actions
test-ubuntu-22.04-cpp GitHub Actions
test-ubuntu-22.04-cpp-20 GitHub Actions
test-ubuntu-22.04-cpp-emscripten GitHub Actions
test-ubuntu-22.04-cpp-no-threading GitHub Actions
test-ubuntu-22.04-python-3 GitHub Actions
test-ubuntu-22.04-python-313-freethreading GitHub Actions
test-ubuntu-24.04-cpp GitHub Actions
test-ubuntu-24.04-cpp-bundled-offline GitHub Actions
test-ubuntu-24.04-cpp-gcc-13-bundled GitHub Actions
test-ubuntu-24.04-cpp-gcc-14 GitHub Actions
test-ubuntu-24.04-cpp-minimal-with-formats GitHub Actions
test-ubuntu-24.04-cpp-thread-sanitizer GitHub Actions
test-ubuntu-24.04-python-3 GitHub Actions

@pitrou
Copy link
Member Author

pitrou commented Dec 13, 2024

It looks like we will have to disable mimalloc for the thread sanitizer test.

@pitrou
Copy link
Member Author

pitrou commented Dec 13, 2024

@github-actions crossbow submit test-ubuntu-24.04-cpp-thread-sanitizer

@pitrou pitrou force-pushed the gh44951-fix-memory-test branch from 8bd302b to 6cb1da6 Compare December 13, 2024 13:49
@github-actions GitHub Actions
Copy link

Revision: 8bd302b

Submitted crossbow builds: ursacomputing/crossbow @ actions-f3f267434f

Task Status
test-ubuntu-24.04-cpp-thread-sanitizer GitHub Actions

@github-actions github-actions bot added awaiting merge Awaiting merge and removed awaiting change review Awaiting change review labels Dec 13, 2024
@pitrou pitrou merged commit f9a6eda into apache:main Dec 13, 2024
36 checks passed
@pitrou pitrou removed the awaiting merge Awaiting merge label Dec 13, 2024
@pitrou pitrou mentioned this pull request Dec 13, 2024
Closed
@pitrou pitrou deleted the gh44951-fix-memory-test branch December 13, 2024 15:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@raulcd raulcd raulcd left review comments

@jorisvandenbossche jorisvandenbossche jorisvandenbossche approved these changes

@assignUser assignUser Awaiting requested review from assignUser assignUser is a code owner

@jonkeane jonkeane Awaiting requested review from jonkeane jonkeane is a code owner

@kou kou Awaiting requested review from kou kou is a code owner

Assignees
No one assigned
Labels
Component: Python
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@pitrou @jorisvandenbossche @raulcd