ARROW-16510: [R] Add bindings for GCS filesystem #13404

nealrichardson · 2022-06-20T15:26:33Z

This adds basic bindings for GcsFileSystem to R, turns it on in the macOS, Windows, and Linux packaging (same handling as ARROW_S3), and basic R tests.

Followups:

Bindings for FromImpersonatedServiceAccount (ARROW-16885)
Set up testbench for fuller tests, like how we do with minio (ARROW-16879)
GcsFileSystem::Make should return Result (ARROW-16884)
Explore auth integration/compatibility with gargle, googleAuthR, etc.: can we pick up the same credentials they use (ARROW-16880)
macOS binary packaging: push dependencies upstream (ARROW-16883)
Windows binary packaging: push dependencies upstream (ARROW-16878)
Update cloud/filesystem documentation (ARROW-16887)

github-actions · 2022-06-20T15:27:20Z

https://issues.apache.org/jira/browse/ARROW-16510

nealrichardson · 2022-06-20T15:27:49Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

        strings
        strings_internal
        symbolize
        synchronization
        throw_delegate
        time
-        time_zone)
+        time_zone
+        wyhash)


Most of the cmake changes here seem to just be differences in sorting (due to my locale?), but the omission of wyhash seems to be legitimately new.

nealrichardson · 2022-06-20T15:28:32Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

    #   sed -e 's;.*/absl_;set_property(TARGET absl::;' \
    #       -e 's/.pc:Requires:/ PROPERTY INTERFACE_LINK_LIBRARIES /' \
-    #       -e 's/ = 20210324,//g' \
-    #       -e 's/ = 20210324//g' \
+    #       -E -e 's/ = 20[0-9]{6},?//g' \


I simplified this regex so that it is robust to future abseil version bumps (at least until 2030!)

nealrichardson · 2022-06-20T15:29:09Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

@@ -3542,18 +3544,6 @@ macro(resolve_dependency_absl)
                   APPEND
                   PROPERTY INTERFACE_LINK_LIBRARIES ${CoreFoundation})
    endif()
-    set_property(TARGET absl::type_traits PROPERTY INTERFACE_LINK_LIBRARIES absl::config)


These are duplicated above so I removed them here; I suspect these were an artifact of a bad merge

nealrichardson · 2022-06-20T15:30:50Z

r/data-raw/codegen.R

@@ -44,7 +44,9 @@ get_exported_functions <- function(decorations, export_tag) {
  out <- decorations %>%
    filter(decoration %in% paste0(export_tag, "::export")) %>%
    mutate(functions = map(context, decor:::parse_cpp_function)) %>%
-    { vec_cbind(., vec_rbind(!!!pull(., functions))) } %>%
+    {


This and the rest of the changes in the file seem to be just styler running on the file (vscode did it on save). Since the code generated after these style changes looks identical (there is no massive diff in the arrowExports.* files), maybe we can remove the styler_exclusion on this file?

r/src/arrow_types.h

r/src/filesystem.cpp

nealrichardson · 2022-06-21T17:30:14Z

I turned on GCS wherever S3 was turned on (so NOT_CRAN=true will build GCS on linux). We should check the build times and see if it's worth the cost or whether it should be a more explicit opt-in. I think we should have it on in the binaries we build though.

nealrichardson · 2022-06-21T18:39:15Z

R debian build is failing with an undefined symbol from absl, a different one from what I saw locally (on macOS):

https://github.com/apache/arrow/runs/6989838112?check_suite_focus=true#step:5:2990

nealrichardson · 2022-06-21T21:21:01Z

@github-actions crossbow submit homebrew-r-autobrew

github-actions · 2022-06-21T21:22:19Z

Revision: df5033b

Submitted crossbow builds: ursacomputing/crossbow @ actions-b4426052af

Task	Status
homebrew-r-autobrew

nealrichardson · 2022-06-21T22:17:47Z

@coryan any ideas about that undefined abseil symbol mentioned above? undefined symbol: _ZN4absl12lts_2021110212RFC3339_fullE

Similar issue in the homebrew build Symbol not found: __ZN4absl12lts_2021110210FormatTimeENS0_11string_viewENS0_4TimeENS0_8TimeZoneE See #13407 (comment)

coryan · 2022-06-21T22:45:10Z

TL;DR; not much of an idea, sorry.

any ideas about that undefined abseil symbol mentioned above?

I assume "above" refers to https://github.com/apache/arrow/runs/6989838112?check_suite_focus=true#step:5:2990 just let me know if I read that wrong.

undefined symbol: _ZN4absl12lts_2021110212RFC3339_fullE

Seems like Abseil is not being linked? The log says:

-- All bundled static libraries: Snappy::snappy;thrift::thrift;jemalloc;mimalloc::mimalloc;lz4::lz4;zstd::libzstd;re2::re2;utf8proc::utf8proc;google-cloud-cpp::storage;google-cloud-cpp::common;aws-cpp-sdk-identity-management;aws-cpp-sdk-sts;aws-cpp-sdk-cognito-identity;aws-cpp-sdk-s3;aws-cpp-sdk-core;AWS::aws-c-event-stream;AWS::aws-checksums;AWS::aws-c-common

That seems to be missing Abseil, but I do not quite understand how the list of bundled dependencies is created.

Similar issue in the homebrew build Symbol not found: __ZN4absl12lts_2021110210FormatTimeENS0_11string_viewENS0_4TimeENS0_8TimeZoneE See #13407 (comment)

Probably a similar problem as above. Less likely, it could be Abseil's sensitivity to C++11 vs. C++17 building (see abseil/abseil-cpp#696), I always think it could be that issue if string_view, variant or any are involved.

nealrichardson · 2022-06-21T23:08:16Z

@coryan thanks, that's helpful actually. I'll have to dig more on the bundled libraries thing. On the Homebrew build, abseil indeed is being built with C++17 by brew (https://github.com/Homebrew/homebrew-core/blob/master/Formula/abseil.rb#L31), and the arrow build is using C++11. What's odd is that we've been depending on it (transitively) for a while because Flight uses grpc (also built C++17 by brew), but I guess google-cloud-cpp hits different parts.

nealrichardson · 2022-06-21T23:43:02Z

Well, that didn't work, there must be some other piece I'm missing:

-- Configuring done
CMake Error at cmake_modules/BuildUtils.cmake:112 (file):
  Error evaluating generator expression:
    $<TARGET_FILE:absl::memory>
  Target "absl::memory" is not an executable or library.
Call Stack (most recent call first):
  src/arrow/CMakeLists.txt:608 (create_merged_static_lib)
CMake Error at cmake_modules/BuildUtils.cmake:112 (file):
  Error evaluating generator expression:
    $<TARGET_FILE:absl::memory>
  Target "absl::memory" is not an executable or library.
Call Stack (most recent call first):
  src/arrow/CMakeLists.txt:608 (create_merged_static_lib)

coryan · 2022-06-21T23:58:40Z

I suspect absl::memory is a header-only library, so it does not have a $<TARGET_FILE>.

coryan · 2022-06-22T16:49:48Z

If you have a VM with abseil installed, you could do something like:

pkg-config --libs absl_memory absl_strings absl_str_format absl_time absl_variant absl_base absl_memory absl_optional absl_span absl_time absl_variant
-L/usr/local/lib64 -labsl_str_format_internal -labsl_bad_optional_access -labsl_time -labsl_civil_time -labsl_strings -labsl_strings_internal -lrt -labsl_base -labsl_spinlock_wait -labsl_int128 -labsl_throw_delegate -labsl_time_zone -labsl_bad_variant_access -labsl_raw_logging_internal -labsl_log_severity

Which (a) removes the header-only libraries, and (b) saves you some whack-a-moling (maybe).

nealrichardson · 2022-06-22T17:01:47Z

If you have a VM with abseil installed, you could do something like:

pkg-config --libs absl_memory absl_strings absl_str_format absl_time absl_variant absl_base absl_memory absl_optional absl_span absl_time absl_variant
-L/usr/local/lib64 -labsl_str_format_internal -labsl_bad_optional_access -labsl_time -labsl_civil_time -labsl_strings -labsl_strings_internal -lrt -labsl_base -labsl_spinlock_wait -labsl_int128 -labsl_throw_delegate -labsl_time_zone -labsl_bad_variant_access -labsl_raw_logging_internal -labsl_log_severity

Which (a) removes the header-only libraries, and (b) saves you some whack-a-moling (maybe).

Yeah I keep thinking I've reached the end (just one more library!) but that would be smarter. I have a local build of abseil already (from the arrow build), didn't think to point pkg-config at it. Thanks for the suggestion!

nealrichardson · 2022-06-22T19:14:21Z

@github-actions crossbow submit homebrew-r-autobrew

github-actions · 2022-06-22T19:15:36Z

Revision: 4d789f0

Submitted crossbow builds: ursacomputing/crossbow @ actions-f35855420a

Task	Status
homebrew-r-autobrew

nealrichardson · 2022-06-22T21:45:00Z

For the R Windows packages, I need to use a static libcurl, and stack overflow tells me that I need to define CURL_STATICLIB for the google-cloud-cpp build. (The thing I just pushed won't work, it's not going to put it in the right place.) I don't see a way I can pass that in to the build macro from the outside (as in, from the cmake command that builds arrow), and I don't think we'd want to hard-code that. Is it possible in cmake to tell if find_package(curl) found a static or shared library? Then I think I could conditionally add that only when appropriate. @kou thoughts?

cc @jeroen in case you have different ideas

jeroen · 2022-06-22T22:03:55Z

I think find_package() is supposed to shell out to pkg-config --cflags libcurl and pkg-config --libs libcurl which should automatically include the appropriate macros and static libraries. For example with rtools40 I see:

$ pkg-config --cflags libcurl
-DCURL_STATICLIB -IC:/rtools40/ucrt64/include

$ pkg-config --libs libcurl
-LC:/rtools40/ucrt64/lib -LD:/a/_temp/msys/msys64/ucrt64/lib -lcurl -lnormaliz -lssh2 -lcrypt32 -lgdi32 -lws2_32 -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32 -lz -lssl -lcrypto -lssl -lcrypto -lws2_32 -lgdi32 -lcrypt32 -lgdi32 -lcrypt32 -lwldap32 -lz -lws2_32

wjones127 · 2022-06-22T22:07:53Z

(I created docs follow up: https://issues.apache.org/jira/browse/ARROW-16887)

wjones127 · 2022-06-22T23:42:58Z

r/configure.win

+  if [ $(cmake_option ARROW_GCS) -eq 1 ]; then
+    PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS -DCURL_STATICLIB"
+  fi


I think we should be able to let pkg-config handle adding the flags and libraries for libcurl:

Suggested change

if [ $(cmake_option ARROW_GCS) -eq 1 ]; then

PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS -DCURL_STATICLIB"

fi

if [ $(cmake_option ARROW_GCS) -eq 1 ]; then

PKG_CFLAGS="$PKG_CFLAGS -DARROW_R_WITH_GCS"

PKG_CONFIG_PACKAGES="$PKG_CONFIG_PACKAGES libcurl"

fi

Nevermind this might not help if libcurl is already statically linked in the arrow binaries.

nealrichardson · 2022-06-25T17:14:09Z

@github-actions crossbow submit -g r

github-actions · 2022-06-25T17:15:19Z

Revision: ee4e8e6

Submitted crossbow builds: ursacomputing/crossbow @ actions-fdfd53e5a1

Task	Status
conda-linux-gcc-py37-cpu-r40
conda-linux-gcc-py37-cpu-r41
conda-osx-clang-py37-r40
conda-osx-clang-py37-r41
conda-win-vs2017-py37-r40
conda-win-vs2017-py37-r41
homebrew-r-autobrew
homebrew-r-brew
test-fedora-r-clang-sanitizer
test-r-arrow-backwards-compatibility
test-r-depsource-bundled
test-r-depsource-system
test-r-dev-duckdb
test-r-devdocs
test-r-gcc-11
test-r-gcc-12
test-r-install-local
test-r-linux-as-cran
test-r-linux-rchk
test-r-linux-valgrind
test-r-minimal-build
test-r-offline-maximal
test-r-offline-minimal
test-r-rhub-debian-gcc-devel-lto-latest
test-r-rhub-debian-gcc-release-custom-ccache
test-r-rhub-ubuntu-gcc-release-latest
test-r-rocker-r-base-latest
test-r-rstudio-r-base-4.1-opensuse153
test-r-rstudio-r-base-4.2-centos7-devtoolset-8
test-r-rstudio-r-base-4.2-focal
test-r-ubuntu-22.04
test-r-versions
test-ubuntu-18.04-r-sanitizer

kou · 2022-06-25T20:07:54Z

MinGW 64 failure is something about boost in compiling the GCS filesystem tests: https://github.com/apache/arrow/runs/7054579528?check_suite_focus=true#step:7:1310

We need a preparation to use boost/process.hpp with Mingw-w64 like https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3_test_util.cc#L24-L38 . We can handle it in a follow-up task.

nealrichardson · 2022-06-25T21:00:46Z

@github-actions crossbow submit test-r-offline-maximal test-r-depsource-system homebrew-r-brew

github-actions · 2022-06-25T21:01:44Z

Revision: 226c31b

Submitted crossbow builds: ursacomputing/crossbow @ actions-5668d3fe2c

Task	Status
homebrew-r-brew
test-r-depsource-system
test-r-offline-maximal

nealrichardson · 2022-06-25T21:20:20Z

MinGW 64 failure is something about boost in compiling the GCS filesystem tests: https://github.com/apache/arrow/runs/7054579528?check_suite_focus=true#step:7:1310

We need a preparation to use boost/process.hpp with Mingw-w64 like https://github.com/apache/arrow/blob/master/cpp/src/arrow/filesystem/s3_test_util.cc#L24-L38 . We can handle it in a follow-up task.

I made ARROW-16906 for that and added a TODO comment in the workflow.

nealrichardson · 2022-06-25T21:20:34Z

@github-actions crossbow submit homebrew-r-brew

github-actions · 2022-06-25T21:21:35Z

Revision: a2a5e87

Submitted crossbow builds: ursacomputing/crossbow @ actions-06d32e61d7

Task	Status
homebrew-r-brew

…ged to master

nealrichardson · 2022-06-25T21:40:29Z

Ok @kou we've resolved all of the builds now, this is ready if you want to give it a final review/+1.

nealrichardson · 2022-06-25T22:27:40Z

One observation: the fully bundled Linux build (R / rhub/debian-gcc-devel:latest) seems to be 10-15 minutes slower with ARROW_GCS=ON. We may want to reconsider turning ARROW_GCS=ON without an explicit request to turn it on in the Linux source build (we can include it in the binaries), but that raises some other UX questions. cc @jonkeane

kou

+1

kou · 2022-06-26T01:02:21Z

cpp/cmake_modules/ThirdpartyToolchain.cmake

  add_library(google-cloud-cpp::storage STATIC IMPORTED)
  set_target_properties(google-cloud-cpp::storage
                        PROPERTIES IMPORTED_LOCATION
                                   "${GOOGLE_CLOUD_CPP_STATIC_LIBRARY_STORAGE}"
                                   INTERFACE_INCLUDE_DIRECTORIES
                                   "${GOOGLE_CLOUD_CPP_INCLUDE_DIR}")
+  # Update this from https://github.com/googleapis/google-cloud-cpp/blob/main/google/cloud/storage/google_cloud_cpp_storage.cmake
  set_property(TARGET google-cloud-cpp::storage
               PROPERTY INTERFACE_LINK_LIBRARIES
                        google-cloud-cpp::common


We can remove this because google-cloud-cpp-rest-internal depends on google-cloud-cpp::common.

I guess so, but since it's copied from upstream, I'd rather leave it there for completeness unless it's harming anything.

cpp/cmake_modules/ThirdpartyToolchain.cmake

kou · 2022-06-26T01:10:21Z

r/configure

+
+  arrow_built_with() {
+    # Function to check cmake options for features
+    grep -i 'set('"$1"' "ON")' $ARROW_OPTS_CMAKE >/dev/null 2>&1


FYI: We can use grep -q ... instead of grep ... >/dev/null 2>&1.

grep -q doesn't suppress stderr though, and I want this to be completely silent

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

This adds basic bindings for GcsFileSystem to R, turns it on in the macOS, Windows, and Linux packaging (same handling as ARROW_S3), and basic R tests. Followups: - Bindings for FromImpersonatedServiceAccount (ARROW-16885) - Set up testbench for fuller tests, like how we do with minio (ARROW-16879) - GcsFileSystem::Make should return Result (ARROW-16884) - Explore auth integration/compatibility with `gargle`, `googleAuthR`, etc.: can we pick up the same credentials they use (ARROW-16880) - macOS binary packaging: push dependencies upstream (ARROW-16883) - Windows binary packaging: push dependencies upstream (ARROW-16878) - Update cloud/filesystem documentation (ARROW-16887) Lead-authored-by: Neal Richardson <neal.p.richardson@gmail.com> Co-authored-by: Sutou Kouhei <kou@clear-code.com> Signed-off-by: Neal Richardson <neal.p.richardson@gmail.com>

github-actions bot added Component: C++ Component: R labels Jun 20, 2022

nealrichardson commented Jun 20, 2022

View reviewed changes

r/src/arrow_types.h Outdated Show resolved Hide resolved

nealrichardson commented Jun 20, 2022

View reviewed changes

r/src/filesystem.cpp Outdated Show resolved Hide resolved

nealrichardson commented Jun 20, 2022

View reviewed changes

r/src/filesystem.cpp Outdated Show resolved Hide resolved

nealrichardson marked this pull request as ready for review June 21, 2022 18:39

nealrichardson mentioned this pull request Jun 21, 2022

ARROW-16869: [CI][C++][Homebrew] Build Apache Arrow with C++17 #13407

Merged

nealrichardson force-pushed the r-gcs branch from 05c6cd0 to 4816aa7 Compare June 22, 2022 12:57

wjones127 reviewed Jun 22, 2022

View reviewed changes

kou and others added 8 commits June 25, 2022 13:13

Fix order

52f6b45

Add missing library

598c665

Add license header

aa059b3

Increase timeout

d843122

Remove needless CURL_STATICLIB check

3192b0a

Upgrade google-cloud-cpp to 1.42.0 to resolve mingw issues

1f9f9de

Add comment for google-cloud-cpp patch

2dbed38

Turn ARROW_GCS back off in mingw C++ workflow

ee4e8e6

nealrichardson force-pushed the r-gcs branch from 5ec3617 to ee4e8e6 Compare June 25, 2022 17:13

Fixes for failing nightly builds

226c31b

nealrichardson added 2 commits June 25, 2022 17:17

Try this for brew

c7a5f12

Add TODO

a2a5e87

Back out brew job change and note TODO. This will pass once it is mer…

5dad768

…ged to master

kou approved these changes Jun 26, 2022

View reviewed changes

Swap order in bundled static libs

ea76c70

Co-authored-by: Sutou Kouhei <kou@cozmixng.org>

nealrichardson merged commit 3ac0959 into apache:master Jun 26, 2022

nealrichardson deleted the r-gcs branch June 26, 2022 13:43

asfimport mentioned this pull request Nov 23, 2022

[R] Add bindings for GCS filesystem #31873

Closed

ARROW-16510: [R] Add bindings for GCS filesystem #13404

ARROW-16510: [R] Add bindings for GCS filesystem #13404

Conversation

nealrichardson commented Jun 20, 2022 • edited Loading

github-actions bot commented Jun 20, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson commented Jun 21, 2022

nealrichardson commented Jun 21, 2022

nealrichardson commented Jun 21, 2022

github-actions bot commented Jun 21, 2022

nealrichardson commented Jun 21, 2022

coryan commented Jun 21, 2022

nealrichardson commented Jun 21, 2022

nealrichardson commented Jun 21, 2022

coryan commented Jun 21, 2022

coryan commented Jun 22, 2022

nealrichardson commented Jun 22, 2022

nealrichardson commented Jun 22, 2022

github-actions bot commented Jun 22, 2022

nealrichardson commented Jun 22, 2022

jeroen commented Jun 22, 2022 • edited Loading

wjones127 commented Jun 22, 2022

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson commented Jun 25, 2022

github-actions bot commented Jun 25, 2022

kou commented Jun 25, 2022

nealrichardson commented Jun 25, 2022

github-actions bot commented Jun 25, 2022

nealrichardson commented Jun 25, 2022

nealrichardson commented Jun 25, 2022

github-actions bot commented Jun 25, 2022

nealrichardson commented Jun 25, 2022

nealrichardson commented Jun 25, 2022

kou left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nealrichardson commented Jun 20, 2022 •

edited

Loading

jeroen commented Jun 22, 2022 •

edited

Loading