Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ARROW-14506: [C++] Conda support for google-cloud-cpp #11916

Closed
wants to merge 18 commits into from
Closed

ARROW-14506: [C++] Conda support for google-cloud-cpp #11916

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
584f3f4
ARROW-14506: [C++] Conda support for google-cloud-cpp
coryan Dec 3, 2021
c0ef3c5
Use mambabuild
xhochy Jan 23, 2022
7f3f567
Update conda configs
xhochy Jan 23, 2022
44d84f5
Add openssl to conda recipe
xhochy Jan 23, 2022
9310edb
Set CMAKE_CXX_STANDARD in conda recipe
xhochy Jan 23, 2022
e00c14d
Update macOS Azure image
xhochy Jan 23, 2022
76ea85a
Remove .yaml from configs
xhochy Jan 23, 2022
7938e2e
Use mambabuild on Windows
xhochy Jan 23, 2022
11a14d9
Update azure-win
xhochy Jan 24, 2022
ad78f04
More activation on windows
xhochy Jan 24, 2022
d766bf7
Update build script for pyarrow
xhochy Jan 24, 2022
54d1062
Update r variant files
xhochy Jan 24, 2022
a34c893
Update r-arrow recipe
xhochy Jan 24, 2022
d87af2a
Fix path to R sources
xhochy Jan 24, 2022
cd79992
Pin aws-sdk-cpp
xhochy Feb 6, 2022
8c4f77a
Add explicit conversion
coryan Feb 6, 2022
366b494
Use mamba to install and update boa
xhochy Feb 6, 2022
057860d
Pin to stable aws-sdk-cpp
xhochy Feb 6, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion ci/conda_env_cpp.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,7 +15,7 @@
# specific language governing permissions and limitations
# under the License.

aws-sdk-cpp
aws-sdk-cpp=1.8.186
benchmark>=1.6.0
boost-cpp>=1.68.0
brotli
Expand All @@ -25,13 +25,17 @@ cmake
gflags
glog
gmock>=1.10.0
google-cloud-cpp>=1.34.0
grpc-cpp>=1.27.3
gtest>=1.10.0
libprotobuf
libutf8proc
lz4-c
make
ninja
# Required by google-cloud-cpp, the Conda package is missing the dependency:
# https://github.com/conda-forge/google-cloud-cpp-feedstock/issues/28
nlohmann_json
pkg-config
python
rapidjson
Expand Down
9 changes: 6 additions & 3 deletions ci/docker/conda-cpp.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,9 +22,6 @@ FROM ${repo}:${arch}-conda
COPY ci/scripts/install_minio.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_minio.sh latest /opt/conda

COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_gcs_testbench.sh default

# install the required conda packages into the test environment
COPY ci/conda_env_cpp.txt \
ci/conda_env_gandiva.txt \
Expand All @@ -37,12 +34,17 @@ RUN mamba install \
valgrind && \
mamba clean --all

# We want to install the GCS testbench using the same Python binary that the Conda code will use.
COPY ci/scripts/install_gcs_testbench.sh /arrow/ci/scripts
RUN /arrow/ci/scripts/install_gcs_testbench.sh default

ENV ARROW_BUILD_TESTS=ON \
ARROW_DATASET=ON \
ARROW_DEPENDENCY_SOURCE=CONDA \
ARROW_FLIGHT=ON \
ARROW_FLIGHT_SQL=ON \
ARROW_GANDIVA=ON \
ARROW_GCS=ON \
ARROW_HOME=$CONDA_PREFIX \
ARROW_ORC=ON \
ARROW_PARQUET=ON \
Expand All @@ -55,6 +57,7 @@ ENV ARROW_BUILD_TESTS=ON \
ARROW_WITH_SNAPPY=ON \
ARROW_WITH_ZLIB=ON \
ARROW_WITH_ZSTD=ON \
CMAKE_CXX_STANDARD=17 \
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this will break OSX for now, so it should only be set for Linux (as it is done here). See the tracking issue conda-forge/clang-compiler-activation-feedstock#17 for that. Can we remove the explicit setting of CMAKE_CXX_STANDARD in CMake instead?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note this will break OSX for now, so it should only be set for Linux (as it is done here).

So this change is doing the "Right Thing"[tm] but maybe accidentally?

See the tracking issue conda-forge/clang-compiler-activation-feedstock#17 for that.

Ack.

Can we remove the explicit setting of CMAKE_CXX_STANDARD in CMake instead?

That is way above my pay grade (for this project). It depends on whether arrow supports any compiler where the default C++ version is < C++11. For example, GCC defaults to C++98 until GCC 6.x, and there is a similar story with Clang.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@pitrou do you have any thoughts on whether we can change the default for CMAKE_CXX_STANDARD?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hmm. I'm not sure I understand the situation entirely, but for now Arrow C++ needs to compile on gcc 4.9 (and perhaps even gcc 4.8, for a subset of Arrow). This is because of the compiler requirements for R packages...

Copy link
Member

@pitrou pitrou Dec 13, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, unless I misunderstand @xhochy 's first message ("it should only be set for Linux (as it is done here)"), it seems there's no real problem here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My message was mainly a heads-up that on OSX, you need to set CMAKE_CXX_STANDARD to 14. This is Linux here, so everything is fine here.

I'm not sure about the general removal of CMAKE_CXX_STANDARD as google-cloud-cpp does also set it: https://github.com/googleapis/google-cloud-cpp/blob/13ec1e946ae1baad6bcae952daf5910649dcfd0a/CMakeLists.txt#L31-L41 Possibly that combination could also be used here?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we already have the following:

if(NOT DEFINED CMAKE_CXX_STANDARD)
  set(CMAKE_CXX_STANDARD 11)
endif()

The only difference AFAICT is that we don't error out if the user explicitly asked for something earlier than C++11, but that must be a really rare case.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@xhochy Can you remind me the reason of CMAKE_CXX_STANDARD=17 here? Also, perhaps explain it in a comment?

GTest_SOURCE=BUNDLED \
PARQUET_BUILD_EXAMPLES=ON \
PARQUET_BUILD_EXECUTABLES=ON \
Expand Down
1 change: 1 addition & 0 deletions ci/scripts/cpp_build.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -135,6 +135,7 @@ cmake \
-DCMAKE_BUILD_TYPE=${ARROW_BUILD_TYPE:-debug} \
-DCMAKE_C_FLAGS="${CFLAGS:-}" \
-DCMAKE_CXX_FLAGS="${CXXFLAGS:-}" \
-DCMAKE_CXX_STANDARD="${CMAKE_CXX_STANDARD:-11}" \
-DCMAKE_INSTALL_LIBDIR=${CMAKE_INSTALL_LIBDIR:-lib} \
-DCMAKE_INSTALL_PREFIX=${CMAKE_INSTALL_PREFIX:-${ARROW_HOME}} \
-DCMAKE_UNITY_BUILD=${CMAKE_UNITY_BUILD:-OFF} \
Expand Down
2 changes: 1 addition & 1 deletion cpp/src/arrow/filesystem/gcsfs.cc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ class GcsInputStream : public arrow::io::InputStream {
stream_.read(reinterpret_cast<char*>(buffer->mutable_data()), nbytes);
ARROW_GCS_RETURN_NOT_OK(stream_.status());
RETURN_NOT_OK(buffer->Resize(stream_.gcount(), true));
return buffer;
return std::shared_ptr<Buffer>(std::move(buffer));
}
//@}

Expand Down
26 changes: 13 additions & 13 deletions ...on10.2numpy1.17python3.7.____cpython.yaml → ...rsion7numpy1.18python3.7.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
aws_sdk_cpp:
- 1.8.186
bzip2:
- '1'
c_compiler:
- gcc
c_compiler_version:
- '9'
- '7'
cdt_name:
- cos6
channel_sources:
- conda-forge,defaults
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler:
Expand All @@ -19,23 +17,25 @@ cuda_compiler_version:
cxx_compiler:
- gxx
cxx_compiler_version:
- '9'
- '7'
docker_image:
- quay.io/condaforge/linux-anvil-cuda:10.2
- quay.io/condaforge/linux-anvil-cos7-cuda:10.2
gflags:
- '2.2'
glog:
- '0.5'
grpc_cpp:
- '1.38'
- '1.42'
libprotobuf:
- '3.16'
- '3.19'
lz4_c:
- 1.9.3
numpy:
- '1.17'
- '1.18'
openssl:
- 1.1.1
orc:
- 1.6.8
- 1.7.2
pin_run_as_build:
bzip2:
max_pin: x
Expand All @@ -49,17 +49,17 @@ pin_run_as_build:
python:
- 3.7.* *_cpython
re2:
- 2021.06.01
- 2021.11.01
snappy:
- '1'
target_platform:
- linux-64
thrift_cpp:
- 0.14.2
- 0.15.0
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- - cuda_compiler_version
- cuda_compiler_version
- cdt_name
- docker_image
- - python
Expand Down
26 changes: 13 additions & 13 deletions ...on10.2numpy1.17python3.8.____cpython.yaml → ...rsion7numpy1.18python3.8.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
aws_sdk_cpp:
- 1.8.186
bzip2:
- '1'
c_compiler:
- gcc
c_compiler_version:
- '9'
- '7'
cdt_name:
- cos6
channel_sources:
- conda-forge,defaults
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler:
Expand All @@ -19,23 +17,25 @@ cuda_compiler_version:
cxx_compiler:
- gxx
cxx_compiler_version:
- '9'
- '7'
docker_image:
- quay.io/condaforge/linux-anvil-cuda:10.2
- quay.io/condaforge/linux-anvil-cos7-cuda:10.2
gflags:
- '2.2'
glog:
- '0.5'
grpc_cpp:
- '1.38'
- '1.42'
libprotobuf:
- '3.16'
- '3.19'
lz4_c:
- 1.9.3
numpy:
- '1.17'
- '1.18'
openssl:
- 1.1.1
orc:
- 1.6.8
- 1.7.2
pin_run_as_build:
bzip2:
max_pin: x
Expand All @@ -49,17 +49,17 @@ pin_run_as_build:
python:
- 3.8.* *_cpython
re2:
- 2021.06.01
- 2021.11.01
snappy:
- '1'
target_platform:
- linux-64
thrift_cpp:
- 0.14.2
- 0.15.0
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- - cuda_compiler_version
- cuda_compiler_version
- cdt_name
- docker_image
- - python
Expand Down
24 changes: 12 additions & 12 deletions ...on10.2numpy1.19python3.9.____cpython.yaml → ...rsion7numpy1.19python3.9.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
aws_sdk_cpp:
- 1.8.186
bzip2:
- '1'
c_compiler:
- gcc
c_compiler_version:
- '9'
- '7'
cdt_name:
- cos6
channel_sources:
- conda-forge,defaults
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler:
Expand All @@ -19,23 +17,25 @@ cuda_compiler_version:
cxx_compiler:
- gxx
cxx_compiler_version:
- '9'
- '7'
docker_image:
- quay.io/condaforge/linux-anvil-cuda:10.2
- quay.io/condaforge/linux-anvil-cos7-cuda:10.2
gflags:
- '2.2'
glog:
- '0.5'
grpc_cpp:
- '1.38'
- '1.42'
libprotobuf:
- '3.16'
- '3.19'
lz4_c:
- 1.9.3
numpy:
- '1.19'
openssl:
- 1.1.1
orc:
- 1.6.8
- 1.7.2
pin_run_as_build:
bzip2:
max_pin: x
Expand All @@ -49,17 +49,17 @@ pin_run_as_build:
python:
- 3.9.* *_cpython
re2:
- 2021.06.01
- 2021.11.01
snappy:
- '1'
target_platform:
- linux-64
thrift_cpp:
- 0.14.2
- 0.15.0
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- - cuda_compiler_version
- cuda_compiler_version
- cdt_name
- docker_image
- - python
Expand Down
28 changes: 14 additions & 14 deletions ...on10.2numpy1.17python3.6.____cpython.yaml → ...sion7numpy1.21python3.10.____cpython.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,15 +1,13 @@
aws_sdk_cpp:
- 1.8.186
bzip2:
- '1'
c_compiler:
- gcc
c_compiler_version:
- '9'
- '7'
cdt_name:
- cos6
channel_sources:
- conda-forge,defaults
- conda-forge
channel_targets:
- conda-forge main
cuda_compiler:
Expand All @@ -19,23 +17,25 @@ cuda_compiler_version:
cxx_compiler:
- gxx
cxx_compiler_version:
- '9'
- '7'
docker_image:
- quay.io/condaforge/linux-anvil-cuda:10.2
- quay.io/condaforge/linux-anvil-cos7-cuda:10.2
gflags:
- '2.2'
glog:
- '0.5'
grpc_cpp:
- '1.38'
- '1.42'
libprotobuf:
- '3.16'
- '3.19'
lz4_c:
- 1.9.3
numpy:
- '1.17'
- '1.21'
openssl:
- 1.1.1
orc:
- 1.6.8
- 1.7.2
pin_run_as_build:
bzip2:
max_pin: x
Expand All @@ -47,19 +47,19 @@ pin_run_as_build:
zlib:
max_pin: x.x
python:
- 3.6.* *_cpython
- 3.10.* *_cpython
re2:
- 2021.06.01
- 2021.11.01
snappy:
- '1'
target_platform:
- linux-64
thrift_cpp:
- 0.14.2
- 0.15.0
zip_keys:
- - c_compiler_version
- cxx_compiler_version
- - cuda_compiler_version
- cuda_compiler_version
- cdt_name
- docker_image
- - python
Expand Down
Loading