Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci: avoid fedora's pkgconf which is slow #7061

Merged
merged 1 commit into from
Jul 26, 2021
Merged

ci: avoid fedora's pkgconf which is slow #7061

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions ci/cloudbuild/builds/cmake-install.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ done < <(find "${INSTALL_PREFIX}" -type f -print0)

for repo_root in "ci/verify_current_targets" "ci/verify_deprecated_targets"; do
out_dir="cmake-out/$(basename "${repo_root}")-out"
rm -f "${out_dir}/CMakeCache.txt"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is so that we can run the cmake-install build multiple times on a machine where this file is persistent (e.g., our --docker builds). The problem is that this file caches the temporary install location from the previous invocation.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For a separate PR: maybe the install location should not be temporary? /h/test-install or cmake-out/test-install?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we want a fully clean install each time. Otherwise, if we removed an artifact in one build, the artifact may still exist on disk and we may not notice that we're still depending on the removed artifact.... well, our CI builds in GCB would notice, but we wouldn't notice on our local machines until we blew away the install directory and rebuilt. Anyway, that's why I originally installed in a new dir every time.

io::log_h2 "Verifying CMake targets in repo root: ${repo_root}"
cmake -GNinja -DCMAKE_PREFIX_PATH="${INSTALL_PREFIX}" \
-S "${repo_root}" -B "${out_dir}" -Wno-dev
Expand Down
22 changes: 19 additions & 3 deletions ci/cloudbuild/dockerfiles/demo-fedora.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ ARG NCPU=4

# ```bash
RUN dnf makecache && \
dnf install -y ccache cmake curl gcc-c++ git make ninja-build \
openssl-devel patch pkgconfig unzip tar wget zip zlib-devel
dnf install -y ccache cmake curl findutils gcc-c++ git make ninja-build \
openssl-devel patch unzip tar wget zip zlib-devel
# ```

# Fedora 34 includes packages for gRPC and Protobuf, but they are not
Expand All @@ -34,11 +34,27 @@ RUN dnf makecache && \
dnf install -y c-ares-devel re2-devel libcurl-devel
# ```

# Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
# when handling `.pc` files with lots of `Requires:` deps, which happens with
# Abseil. If you plan to use `pkg-config` with any of the installed artifacts,
# you may want to use a recent version of the standard `pkg-config` binary. If
# not, `dnf install pkgconfig` should work.

# ```bash
WORKDIR /var/tmp/build/pkg-config-cpp
RUN curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
make install && \
ldconfig
# ```

# The following steps will install libraries and tools in `/usr/local`. By
# default pkg-config does not search in these directories.

# ```bash
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig
# ```

# #### Abseil
Expand Down
17 changes: 15 additions & 2 deletions ci/cloudbuild/dockerfiles/fedora-34.Dockerfile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ RUN dnf makecache && \
cmake diffutils doxygen findutils gcc-c++ git \
lcov libcxx-devel libcxxabi-devel \
libasan libubsan libtsan libcurl-devel make ninja-build \
openssl-devel patch pkgconfig python python3.8 \
openssl-devel patch python python3.8 \
python-pip tar unzip w3m wget which zip zlib-devel

# Sets root's password to the empty string to enable users to get a root shell
Expand All @@ -38,6 +38,20 @@ RUN dnf makecache && dnf install -y python3-devel
RUN pip3 install --upgrade pip
RUN pip3 install setuptools wheel

# Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
# when handling `.pc` files with lots of `Requires:` deps, which happens with
# Abseil, so we use the normal `pkg-config` binary, which seems to not suffer
# from this bottleneck. For more details see
# https://github.com/googleapis/google-cloud-cpp/issues/7052
WORKDIR /var/tmp/build/pkg-config-cpp
RUN curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
make install && \
ldconfig
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig

# Install Abseil, remove the downloaded files and the temporary artifacts
# after a successful build to keep the image smaller (and with fewer layers)
WORKDIR /var/tmp/build
Expand Down Expand Up @@ -170,7 +184,6 @@ RUN dnf makecache && dnf install -y java-latest-openjdk-devel

# Some of the above libraries may have installed in /usr/local, so make sure
# those library directories will be found.
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
RUN ldconfig /usr/local/lib*

RUN curl -o /usr/bin/bazelisk -sSL "https://github.com/bazelbuild/bazelisk/releases/download/v1.10.1/bazelisk-linux-amd64" && \
Expand Down
22 changes: 19 additions & 3 deletions doc/packaging.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,8 @@ Install the minimal development tools:

```bash
sudo dnf makecache && \
sudo dnf install -y ccache cmake curl gcc-c++ git make ninja-build \
openssl-devel patch pkgconfig unzip tar wget zip zlib-devel
sudo dnf install -y ccache cmake curl findutils gcc-c++ git make ninja-build \
openssl-devel patch unzip tar wget zip zlib-devel
```

Fedora 34 includes packages for gRPC and Protobuf, but they are not
Expand All @@ -224,11 +224,27 @@ sudo dnf makecache && \
sudo dnf install -y c-ares-devel re2-devel libcurl-devel
```

Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
when handling `.pc` files with lots of `Requires:` deps, which happens with
Abseil. If you plan to use `pkg-config` with any of the installed artifacts,
you may want to use a recent version of the standard `pkg-config` binary. If
not, `sudo dnf install pkgconfig` should work.

```bash
mkdir -p $HOME/Downloads/pkg-config-cpp && cd $HOME/Downloads/pkg-config-cpp
curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
sudo make install && \
sudo ldconfig
```

The following steps will install libraries and tools in `/usr/local`. By
default pkg-config does not search in these directories.

```bash
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig
```

#### Abseil
Expand Down