Skip to content

Commit

Permalink
ci: avoid fedora's pkgconf which is slow
Browse files Browse the repository at this point in the history
Fixes: googleapis#7052

This avoids Fedora's `pkgconf` (a drop-in replacement for `pkg-config`)
because it's too slow when dealing with `.pc` files with lots of
`Requires:` deps, which we have with Abseil. Instead, we use the normal
`pkg-config` binary, which seems to work better.

After this PR, we should be able to upgrade to gRPC 1.39.0 (`git revert
ba41d5a`)
  • Loading branch information
devjgm committed Jul 26, 2021
1 parent b10aadd commit e525827
Show file tree
Hide file tree
Showing 4 changed files with 54 additions and 8 deletions.
1 change: 1 addition & 0 deletions ci/cloudbuild/builds/cmake-install.sh
Original file line number Diff line number Diff line change
Expand Up @@ -149,6 +149,7 @@ done < <(find "${INSTALL_PREFIX}" -type f -print0)

for repo_root in "ci/verify_current_targets" "ci/verify_deprecated_targets"; do
out_dir="cmake-out/$(basename "${repo_root}")-out"
rm -f "${out_dir}/CMakeCache.txt"
io::log_h2 "Verifying CMake targets in repo root: ${repo_root}"
cmake -GNinja -DCMAKE_PREFIX_PATH="${INSTALL_PREFIX}" \
-S "${repo_root}" -B "${out_dir}" -Wno-dev
Expand Down
22 changes: 19 additions & 3 deletions ci/cloudbuild/dockerfiles/demo-fedora.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -21,8 +21,8 @@ ARG NCPU=4

# ```bash
RUN dnf makecache && \
dnf install -y ccache cmake curl gcc-c++ git make ninja-build \
openssl-devel patch pkgconfig unzip tar wget zip zlib-devel
dnf install -y ccache cmake curl findutils gcc-c++ git make ninja-build \
openssl-devel patch unzip tar wget zip zlib-devel
# ```

# Fedora 34 includes packages for gRPC and Protobuf, but they are not
Expand All @@ -34,11 +34,27 @@ RUN dnf makecache && \
dnf install -y c-ares-devel re2-devel libcurl-devel
# ```

# Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
# when handling `.pc` files with lots of `Requires:` deps, which happens with
# Abseil. If you plan to use `pkg-config` with any of the installed artifacts,
# you may want to use a recent version of the standard `pkg-config` binary. If
# not, `dnf install pkgconfig` should work.

# ```bash
WORKDIR /var/tmp/build/pkg-config-cpp
RUN curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
make install && \
ldconfig
# ```

# The following steps will install libraries and tools in `/usr/local`. By
# default pkg-config does not search in these directories.

# ```bash
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig
# ```

# #### Abseil
Expand Down
17 changes: 15 additions & 2 deletions ci/cloudbuild/dockerfiles/fedora-34.Dockerfile
Original file line number Diff line number Diff line change
Expand Up @@ -24,7 +24,7 @@ RUN dnf makecache && \
cmake diffutils doxygen findutils gcc-c++ git \
lcov libcxx-devel libcxxabi-devel \
libasan libubsan libtsan libcurl-devel make ninja-build \
openssl-devel patch pkgconfig python python3.8 \
openssl-devel patch python python3.8 \
python-pip tar unzip w3m wget which zip zlib-devel

# Sets root's password to the empty string to enable users to get a root shell
Expand All @@ -38,6 +38,20 @@ RUN dnf makecache && dnf install -y python3-devel
RUN pip3 install --upgrade pip
RUN pip3 install setuptools wheel

# Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
# when handling `.pc` files with lots of `Requires:` deps, which happens with
# Abseil, so we use the normal `pkg-config` binary, which seems to not suffer
# from this bottleneck. For more details see
# https://github.com/googleapis/google-cloud-cpp/issues/7052
WORKDIR /var/tmp/build/pkg-config-cpp
RUN curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
make install && \
ldconfig
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig

# Install Abseil, remove the downloaded files and the temporary artifacts
# after a successful build to keep the image smaller (and with fewer layers)
WORKDIR /var/tmp/build
Expand Down Expand Up @@ -170,7 +184,6 @@ RUN dnf makecache && dnf install -y java-latest-openjdk-devel

# Some of the above libraries may have installed in /usr/local, so make sure
# those library directories will be found.
ENV PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
RUN ldconfig /usr/local/lib*

RUN curl -o /usr/bin/bazelisk -sSL "https://github.com/bazelbuild/bazelisk/releases/download/v1.10.1/bazelisk-linux-amd64" && \
Expand Down
22 changes: 19 additions & 3 deletions doc/packaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,8 +211,8 @@ Install the minimal development tools:

```bash
sudo dnf makecache && \
sudo dnf install -y ccache cmake curl gcc-c++ git make ninja-build \
openssl-devel patch pkgconfig unzip tar wget zip zlib-devel
sudo dnf install -y ccache cmake curl findutils gcc-c++ git make ninja-build \
openssl-devel patch unzip tar wget zip zlib-devel
```

Fedora 34 includes packages for gRPC and Protobuf, but they are not
Expand All @@ -224,11 +224,27 @@ sudo dnf makecache && \
sudo dnf install -y c-ares-devel re2-devel libcurl-devel
```

Fedora's version of `pkg-config` (https://github.com/pkgconf/pkgconf) is slow
when handling `.pc` files with lots of `Requires:` deps, which happens with
Abseil. If you plan to use `pkg-config` with any of the installed artifacts,
you may want to use a recent version of the standard `pkg-config` binary. If
not, `sudo dnf install pkgconfig` should work.

```bash
mkdir -p $HOME/Downloads/pkg-config-cpp && cd $HOME/Downloads/pkg-config-cpp
curl -sSL https://pkgconfig.freedesktop.org/releases/pkg-config-0.29.2.tar.gz | \
tar -xzf - --strip-components=1 && \
./configure --with-internal-glib && \
make -j ${NCPU:-4} && \
sudo make install && \
sudo ldconfig
```

The following steps will install libraries and tools in `/usr/local`. By
default pkg-config does not search in these directories.

```bash
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig
export PKG_CONFIG_PATH=/usr/local/lib64/pkgconfig:/usr/local/lib/pkgconfig:/usr/lib64/pkgconfig
```

#### Abseil
Expand Down

0 comments on commit e525827

Please sign in to comment.