Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Refactor setup scripts #10670

Open
wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Refactor setup scripts #10670

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 5 additions & 1 deletion .github/workflows/linux-build.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,8 @@ on:
- "CMake/**"
- "third_party/**"
- "scripts/setup-ubuntu.sh"
- "scripts/setup-common.sh"
- "scripts/setup-versions.sh"
- "scripts/setup-helper-functions.sh"
- ".github/workflows/linux-build.yml"

Expand All @@ -36,6 +38,8 @@ on:
- "CMake/**"
- "third_party/**"
- "scripts/setup-ubuntu.sh"
- "scripts/setup-common.sh"
- "scripts/setup-versions.sh"
- "scripts/setup-helper-functions.sh"
- ".github/workflows/linux-build.yml"

Expand Down Expand Up @@ -171,7 +175,7 @@ jobs:
MAKEFLAGS: "NUM_THREADS=8 MAX_HIGH_MEM_JOBS=4 MAX_LINK_JOBS=3"
EXTRA_CMAKE_FLAGS: "-DVELOX_ENABLE_ARROW=ON -DVELOX_ENABLE_PARQUET=ON"
run: |
make debug
make debug

- name: CCache after
run: |
Expand Down
6 changes: 2 additions & 4 deletions .github/workflows/macos.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -63,9 +63,7 @@ jobs:
env:
HOMEBREW_NO_INSTALLED_DEPENDENTS_CHECK: "TRUE"
run: |
source scripts/setup-macos.sh
brew install $MACOS_BUILD_DEPS $MACOS_VELOX_DEPS

scripts/setup-macos.sh
majetideepak marked this conversation as resolved.
Show resolved Hide resolved
echo "NJOBS=`sysctl -n hw.ncpu`" >> $GITHUB_ENV
brew unlink protobuf || echo "protobuf not installed"
brew link --force protobuf@21
Expand All @@ -78,7 +76,7 @@ jobs:

- name: Configure Build
env:
folly_SOURCE: BUNDLED #brew folly does not have int128
fmt_SOURCE: BUNDLED #picked up brew fmt (from ccache) is too new
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Folly doesn't need to be force bundled because it is not coming from brew anymore but is installed by the setup script.
It uses fmt as bundled because the fmt version is too new. This need to be undone once the build script picks up the installed version via the setup script (and not the one present in homebrew from some dependency).

run: |
ccache -sz -M 5Gi
cmake \
Expand Down
4 changes: 4 additions & 0 deletions .github/workflows/scheduled.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ on:
- "CMake/**"
- "third_party/**"
- "scripts/setup-ubuntu.sh"
- "scripts/setup-common.sh"
- "scripts/setup-versions.sh"
- "scripts/setup-helper-functions.sh"
- ".github/workflows/scheduled.yml"

Expand All @@ -36,6 +38,8 @@ on:
- "CMake/**"
- "third_party/**"
- "scripts/setup-ubuntu.sh"
- "scripts/setup-common.sh"
- "scripts/setup-versions.sh"
- "scripts/setup-helper-functions.sh"
- ".github/workflows/scheduled.yml"

Expand Down
6 changes: 4 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -132,11 +132,13 @@ libraries. Once you have checked out Velox, you can setup and build like so:

```shell
$ ./scripts/setup-centos9.sh
$ ./scripts/setup-adapters.sh
$ ./scripts/setup-centos9.sh install_adapters
$ make
```

Note that `setup-adapters.sh` supports MacOS and Ubuntu 20.04 or later.
Note that the `install_adapters` command is available for the supported MacOS and
Ubuntu (20.04 or later) scripts. Individual adapters can be installed by specifying
the individual install command, e.g. `setup-centos9.sh install_aws`.

### Using Clang on Linux

Expand Down
9 changes: 6 additions & 3 deletions scripts/adapters.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -15,15 +15,18 @@
ARG image=ghcr.io/facebookincubator/velox-dev:centos9
FROM $image

COPY scripts/setup-adapters.sh /
COPY scripts/setup-centos9.sh /
COPY scripts/setup-common.sh /
COPY scripts/setup-versions.sh /
COPY scripts/setup-helper-functions.sh /
RUN mkdir build && ( cd build && source /opt/rh/gcc-toolset-12/enable && \
bash /setup-adapters.sh ) && rm -rf build && dnf remove -y conda && dnf clean all
bash /setup-centos9.sh install_adapters ) && rm -rf build && dnf remove -y conda && dnf clean all

# install miniforge
RUN curl -L -o /tmp/miniforge.sh https://github.com/conda-forge/miniforge/releases/download/23.11.0-0/Mambaforge-23.11.0-0-Linux-x86_64.sh && \
bash /tmp/miniforge.sh -b -p /opt/miniforge && \
rm /tmp/miniforge.sh
ENV PATH=/opt/miniforge/condabin:${PATH}
ENV PATH=/opt/miniforge/condabin:${PATH}

# install test dependencies
RUN mamba create -y --name adapters python=3.8
Expand Down
2 changes: 2 additions & 0 deletions scripts/centos.dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,8 @@ ARG image=quay.io/centos/centos:stream9
FROM $image

COPY scripts/setup-helper-functions.sh /
COPY scripts/setup-versions.sh /
COPY scripts/setup-common.sh /
COPY scripts/setup-centos9.sh /
# The removal of the build dir has to happen in the same layer as the build
# to minimize the image size. gh & jq are required for CI
Expand Down
231 changes: 0 additions & 231 deletions scripts/setup-adapters.sh
View file
Open in desktop

This file was deleted.

Loading
Loading