GH-41922: [CI][C++] Update Minio version #44225

pitrou · 2024-09-25T14:26:12Z

We were using a rather old version of Minio for CI tests.

Update the Minio version and ensure the S3FileSystem implementation passes all the tests with it.

GitHub Issue: [CI][C++][Python] Update minio version #41922

github-actions · 2024-09-25T14:26:46Z

⚠️ GitHub issue #41922 has been automatically assigned in GitHub to PR creator.

pitrou · 2024-09-25T16:59:27Z

@github-actions crossbow submit -g cpp -g python -g wheel

pitrou · 2024-09-25T17:20:47Z

So, I don't understand why all Crossbow conda-based jobs above have failed with a weird CMake error, while the GHA conda-based jobs succeed. The error message doesn't ring a bell to me:

CMake Error at cmake_modules/ThirdpartyToolchain.cmake:952 (find_program):
  Could not find EP_CMAKE_RANLIB using the following names: :
Call Stack (most recent call first):
  CMakeLists.txt:545 (include)

@kou Would you be able to advise?

kou · 2024-09-25T21:30:27Z

It seems that conda stopped setting AR and RANLIB:

https://github.com/ursacomputing/crossbow/actions/runs/11037675823/job/30659122209#step:7:1151

+ export 'ARROW_CMAKE_ARGS= -DCMAKE_AR= -DCMAKE_RANLIB='
+ ARROW_CMAKE_ARGS=' -DCMAKE_AR= -DCMAKE_RANLIB='

How about the following?

diff --git a/ci/scripts/cpp_build.sh b/ci/scripts/cpp_build.sh
index bc2bba915f..ac0ba2bf72 100755
--- a/ci/scripts/cpp_build.sh
+++ b/ci/scripts/cpp_build.sh
@@ -34,7 +34,13 @@ if [ ! -z "${CONDA_PREFIX}" ] && [ "${ARROW_EMSCRIPTEN:-OFF}" = "OFF" ]; then
   echo -e "===\n=== Conda environment for build\n==="
   conda list
 
-  export ARROW_CMAKE_ARGS="${ARROW_CMAKE_ARGS} -DCMAKE_AR=${AR} -DCMAKE_RANLIB=${RANLIB}"
+  if [ -n "${AR}" ]; then
+    ARROW_CMAKE_ARGS+=" -DCMAKE_AR=${AR}"
+  fi
+  if [ -n "${RANLIB}" ]; then
+    ARROW_CMAKE_ARGS+=" -DCMAKE_RANLIB=${RANLIB}"
+  fi
+  export ARROW_CMAKE_ARGS
   export ARROW_GANDIVA_PC_CXX_FLAGS=$(echo | ${CXX} -E -Wp,-v -xc++ - 2>&1 | grep '^ ' | awk '{print "-isystem;" substr($1, 1)}' | tr '\n' ';')
 elif [ -x "$(command -v xcrun)" ]; then
   export ARROW_GANDIVA_PC_CXX_FLAGS="-isysroot;$(xcrun --show-sdk-path)"

The GHA conda-based jobs use cached conda environment. So the conda change isn't affected for now.

pitrou · 2024-09-26T08:34:46Z

@github-actions crossbow submit -g cpp -g python

github-actions · 2024-09-26T08:38:50Z

Revision: 90c07b3

Submitted crossbow builds: ursacomputing/crossbow @ actions-299f61e3ea

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-20.04-cpp-minimal-with-formats
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-thread-sanitizer
test-ubuntu-24.04-python-3

pitrou · 2024-09-26T09:14:19Z

@github-actions crossbow submit -g wheel

pitrou · 2024-09-26T09:14:42Z

Your suggested fix looks good, @kou !

github-actions · 2024-09-26T09:17:17Z

Revision: 90c07b3

Submitted crossbow builds: ursacomputing/crossbow @ actions-66b8892afd

Task	Status
python-sdist
wheel-macos-monterey-cp310-cp310-arm64
wheel-macos-monterey-cp311-cp311-arm64
wheel-macos-monterey-cp312-cp312-arm64
wheel-macos-monterey-cp313-cp313-arm64
wheel-macos-monterey-cp313-cp313t-arm64
wheel-macos-monterey-cp39-cp39-arm64
wheel-macos-ventura-cp310-cp310-amd64
wheel-macos-ventura-cp311-cp311-amd64
wheel-macos-ventura-cp312-cp312-amd64
wheel-macos-ventura-cp313-cp313-amd64
wheel-macos-ventura-cp313-cp313t-amd64
wheel-macos-ventura-cp39-cp39-amd64
wheel-manylinux-2-28-cp310-cp310-amd64
wheel-manylinux-2-28-cp310-cp310-arm64
wheel-manylinux-2-28-cp311-cp311-amd64
wheel-manylinux-2-28-cp311-cp311-arm64
wheel-manylinux-2-28-cp312-cp312-amd64
wheel-manylinux-2-28-cp312-cp312-arm64
wheel-manylinux-2-28-cp313-cp313-amd64
wheel-manylinux-2-28-cp313-cp313-arm64
wheel-manylinux-2-28-cp313-cp313t-amd64
wheel-manylinux-2-28-cp313-cp313t-arm64
wheel-manylinux-2-28-cp39-cp39-amd64
wheel-manylinux-2-28-cp39-cp39-arm64
wheel-manylinux-2014-cp310-cp310-amd64
wheel-manylinux-2014-cp310-cp310-arm64
wheel-manylinux-2014-cp311-cp311-amd64
wheel-manylinux-2014-cp311-cp311-arm64
wheel-manylinux-2014-cp312-cp312-amd64
wheel-manylinux-2014-cp312-cp312-arm64
wheel-manylinux-2014-cp313-cp313-amd64
wheel-manylinux-2014-cp313-cp313-arm64
wheel-manylinux-2014-cp313-cp313t-amd64
wheel-manylinux-2014-cp313-cp313t-arm64
wheel-manylinux-2014-cp39-cp39-amd64
wheel-manylinux-2014-cp39-cp39-arm64
wheel-windows-cp310-amd64
wheel-windows-cp311-amd64
wheel-windows-cp312-amd64
wheel-windows-cp313-amd64
wheel-windows-cp39-amd64

kou · 2024-09-26T09:20:25Z

ci/conda_env_gandiva.txt

-clang>=11
-llvmdev>=11
+clang>=11,<19
+llvmdev>=11,<19


FYI: We can remove this by merging GH-44233.

@kou Which one would you rather merge first?

I've merged GH-44233.
I'll update this branch.

kou

+1

felipecrv · 2024-09-30T14:47:19Z

cpp/src/arrow/filesystem/s3fs.cc

  template <typename Error>
-  void SaveBackend(const Aws::Client::AWSError<Error>& error) {
+  S3Backend GetOrSetBackend(const Aws::Client::AWSError<Error>& error) {


GetOrSet is a very weird name. What about SaveBackendIfAbsent or SaveBackendFromErrorIfAbsent?

I don't know, I got the impression that it was a common name in caching patterns:
https://www.google.com/search?client=firefox-b-d&q=%22get_or_set%22

But more importantly, I wanted to stress that this method is also used to retrieve the cached backend. The "Save" proposal doesn't do that (which is why I renamed the method in this PR).

felipecrv · 2024-09-30T16:58:31Z

cpp/src/arrow/filesystem/s3fs_test.cc

@@ -1005,7 +1005,8 @@ TEST_F(TestS3FS, CreateDir) {
  AssertFileInfo(fs_.get(), "bucket/newdir/newsub/newsubsub", FileType::Directory);

  // Existing "file", should fail
-  ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile"));
+  ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile", /*recursive=*/false));
+  ASSERT_RAISES(IOError, fs_->CreateDir("bucket/somefile", /*recursive=*/true));


Someday the filesystem tests could be unified. I was very careful with directory semantics in the azure fs implementations. Testing all cases to ensure they behave like Linux system calls.

Yes, we could open a new issue to improve the generic filesystem tests. Would you like to do that?

Would you like to do that?

No. Because I know that the other filesystems diverge and these details have leaked into user code (i.e. they rely on the broken semantics).

Fair enough. Usually there's a tension into enforcing consistent semantics accross filesystems, and trying to preserve performance when consistency implies more IO roundtrips to a remote host. If there are inconsistencies that are not motivated by performance, they should be investigated.

S3 finally supports Conditional Writes. That means a lot of the check-before-create sequences you need to implement the filesystem semantics can be made much faster now.

https://aws.amazon.com/about-aws/whats-new/2024/08/amazon-s3-conditional-writes/

Ok, I opened #44281 for it

pitrou · 2024-10-01T08:18:58Z

@github-actions crossbow submit -g cpp -g python

github-actions · 2024-10-01T08:22:38Z

Revision: ca25668

Submitted crossbow builds: ursacomputing/crossbow @ actions-1eca55fdde

Task	Status
example-cpp-minimal-build-static
example-cpp-minimal-build-static-system-dependency
example-cpp-tutorial
example-python-minimal-build-fedora-conda
example-python-minimal-build-ubuntu-venv
test-alpine-linux-cpp
test-build-cpp-fuzz
test-conda-cpp
test-conda-cpp-valgrind
test-conda-python-3.10
test-conda-python-3.10-cython2
test-conda-python-3.10-hdfs-2.9.2
test-conda-python-3.10-hdfs-3.2.1
test-conda-python-3.10-pandas-latest-numpy-latest
test-conda-python-3.10-substrait
test-conda-python-3.11
test-conda-python-3.11-dask-latest
test-conda-python-3.11-dask-upstream_devel
test-conda-python-3.11-hypothesis
test-conda-python-3.11-pandas-latest-numpy-1.26
test-conda-python-3.11-pandas-latest-numpy-latest
test-conda-python-3.11-pandas-nightly-numpy-nightly
test-conda-python-3.11-pandas-upstream_devel-numpy-nightly
test-conda-python-3.11-spark-master
test-conda-python-3.12
test-conda-python-3.12-cpython-debug
test-conda-python-3.9
test-conda-python-3.9-pandas-1.1.3-numpy-1.19.5
test-conda-python-emscripten
test-cuda-cpp-ubuntu-20.04-cuda-11.2.2
test-cuda-cpp-ubuntu-22.04-cuda-11.7.1
test-cuda-python-ubuntu-22.04-cuda-11.7.1
test-debian-12-cpp-amd64
test-debian-12-cpp-i386
test-debian-12-python-3-amd64
test-debian-12-python-3-i386
test-fedora-39-cpp
test-fedora-39-python-3
test-ubuntu-20.04-cpp
test-ubuntu-20.04-cpp-bundled
test-ubuntu-22.04-cpp
test-ubuntu-22.04-cpp-20
test-ubuntu-22.04-cpp-emscripten
test-ubuntu-22.04-cpp-no-threading
test-ubuntu-22.04-python-3
test-ubuntu-22.04-python-313-freethreading
test-ubuntu-24.04-cpp
test-ubuntu-24.04-cpp-gcc-13-bundled
test-ubuntu-24.04-cpp-gcc-14
test-ubuntu-24.04-cpp-minimal-with-formats
test-ubuntu-24.04-cpp-thread-sanitizer
test-ubuntu-24.04-python-3

We were using a rather old version of Minio for CI tests. Update the Minio version and ensure the S3FileSystem implementation passes all the tests with it.

pitrou · 2024-10-01T13:24:13Z

@github-actions crossbow submit example-python-minimal-build-fedora-conda

github-actions · 2024-10-01T13:26:28Z

Revision: cce7a92

Submitted crossbow builds: ursacomputing/crossbow @ actions-d81681459f

Task	Status
example-python-minimal-build-fedora-conda

pitrou · 2024-10-01T14:46:55Z

The CI failures are unfortunate but they are also unrelated. I'll merge.

conbench-apache-arrow · 2024-10-01T22:21:49Z

After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6b59098.

There was 1 benchmark result indicating a performance regression:

Commit Run on amd64-m5-4xlarge-linux at 2024-10-01 16:24:48Z
- partitioned-dataset-filter (R) with dataset=dataset-taxi-parquet, language=R, query=vignette

The full Conbench report has more details. It also includes information about 10 possible false positives for unstable benchmarks that are known to sometimes produce them.

github-actions bot added Component: C++ awaiting review Awaiting review labels Sep 25, 2024

pitrou force-pushed the gh41922-update-minio branch from 8c63110 to aefba96 Compare September 25, 2024 14:37

github-actions bot added the Component: Python label Sep 25, 2024

pitrou force-pushed the gh41922-update-minio branch from 6fa9b17 to 16b9784 Compare September 25, 2024 15:05

pitrou mentioned this pull request Sep 25, 2024

GH-43535: [C++] Support the AWS S3 SSE-C encryption #43601

Merged

This comment was marked as outdated.

Sign in to view

pitrou force-pushed the gh41922-update-minio branch from bd7f1fa to 90c07b3 Compare September 26, 2024 08:34

pitrou marked this pull request as ready for review September 26, 2024 09:14

pitrou requested review from assignUser, jonkeane, kou and raulcd as code owners September 26, 2024 09:14

kou reviewed Sep 26, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting review Awaiting review labels Sep 26, 2024

kou force-pushed the gh41922-update-minio branch from 90c07b3 to 486bcc8 Compare September 27, 2024 02:36

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Sep 27, 2024

kou approved these changes Sep 27, 2024

View reviewed changes

github-actions bot removed the awaiting change review Awaiting change review label Sep 27, 2024

felipecrv reviewed Sep 30, 2024

View reviewed changes

github-actions bot added awaiting changes Awaiting changes and removed awaiting merge Awaiting merge labels Sep 30, 2024

pitrou force-pushed the gh41922-update-minio branch from 850eb26 to ca25668 Compare October 1, 2024 08:15

github-actions bot added awaiting change review Awaiting change review and removed awaiting changes Awaiting changes labels Oct 1, 2024

pitrou and others added 10 commits October 1, 2024 15:23

apacheGH-41922: [CI][C++] Update Minio version

cdda080

We were using a rather old version of Minio for CI tests. Update the Minio version and ensure the S3FileSystem implementation passes all the tests with it.

Avoid LLVM 19

081df4c

Debug

4229bef

Use different health check API

bdf86d8

Undo debug

53ef625

Try to solve mc admin failures

cb04092

Finally (?) fix mc issue

16e5fb0

Try suggested fix

7e7fa91

Revert LLVM pin

0357c5f

Workaround for Windows (untested, relying on CI :-))

cce7a92

pitrou force-pushed the gh41922-update-minio branch from ca25668 to cce7a92 Compare October 1, 2024 13:23

pitrou merged commit 6b59098 into apache:main Oct 1, 2024
37 of 40 checks passed

pitrou removed the awaiting change review Awaiting change review label Oct 1, 2024

pitrou mentioned this pull request Oct 1, 2024

[CI][C++][Python] Update minio version #41922

Closed

pitrou deleted the gh41922-update-minio branch October 1, 2024 14:50

github-actions bot added the awaiting changes Awaiting changes label Oct 1, 2024

pitrou mentioned this pull request Oct 1, 2024

[C++] Use S3 conditional writes to avoid some explicit checks #44281

Open

h-vetinari mentioned this pull request Oct 17, 2024

[Python][CI] Compatibility with newer minio #37692

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

GH-41922: [CI][C++] Update Minio version #44225

GH-41922: [CI][C++] Update Minio version #44225

pitrou commented Sep 25, 2024 •

edited

Loading

github-actions bot commented Sep 25, 2024

pitrou commented Sep 25, 2024

This comment was marked as outdated.

pitrou commented Sep 25, 2024 •

edited

Loading

kou commented Sep 25, 2024

pitrou commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

pitrou commented Sep 26, 2024

pitrou commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

kou Sep 26, 2024

pitrou Sep 26, 2024

kou Sep 27, 2024

kou left a comment

felipecrv Sep 30, 2024

pitrou Sep 30, 2024

pitrou Oct 1, 2024

felipecrv Sep 30, 2024

pitrou Sep 30, 2024

felipecrv Oct 1, 2024

pitrou Oct 1, 2024

felipecrv Oct 1, 2024

pitrou Oct 1, 2024

pitrou commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

pitrou commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

pitrou commented Oct 1, 2024

conbench-apache-arrow bot commented Oct 1, 2024

GH-41922: [CI][C++] Update Minio version #44225

GH-41922: [CI][C++] Update Minio version #44225

Conversation

pitrou commented Sep 25, 2024 • edited Loading

github-actions bot commented Sep 25, 2024

pitrou commented Sep 25, 2024

This comment was marked as outdated.

pitrou commented Sep 25, 2024 • edited Loading

kou commented Sep 25, 2024

pitrou commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

pitrou commented Sep 26, 2024

pitrou commented Sep 26, 2024

github-actions bot commented Sep 26, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pitrou commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

pitrou commented Oct 1, 2024

github-actions bot commented Oct 1, 2024

pitrou commented Oct 1, 2024

conbench-apache-arrow bot commented Oct 1, 2024

pitrou commented Sep 25, 2024 •

edited

Loading

pitrou commented Sep 25, 2024 •

edited

Loading