Build client library for only pubsub

[aks-zero]

We are using the pubsub client library to send messages from a C++ application to pubsub topics in GCP and the functionality is working as expected.

My question is about the build process. We use cmake and would like to build only the pubsub client library as that's the only service we use. However looking at the build logs it seems like it's building and installing a lot more than that e.g. bigtable, bigquery, dialogflow etc. even though we have enabled the -DGOOGLE_CLOUD_CPP_ENABLE=pubsub flag as below (the code is from Dockerfile, but the end result is same when I run cmake manually on the console). Can someone please review and suggest how can we improve this?

ARG GOOGLE_CLIENT_CPP_BRANCH=v2.3.0
RUN git clone -b ${GOOGLE_CLIENT_CPP_BRANCH} --depth 1 https://github.com/googleapis/google-cloud-cpp.git \
    && cd google-cloud-cpp \
    && cmake -H. -Bcmake-out \
    -DBUILD_TESTING=OFF -DGOOGLE_CLOUD_CPP_ENABLE_EXAMPLES=OFF -DCMAKE_BUILD_TYPE="Release" -DCMAKE_POSITION_INDEPENDENT_CODE=ON -DCMAKE_CXX_STANDARD=17 -DGOOGLE_CLOUD_CPP_ENABLE=pubsub \
    && cmake --build cmake-out --target install -- -j $(nproc)

[dbolduc]

Hey, what you are doing with -DGOOGLE_CLOUD_CPP_ENABLE=pubsub looks right to me.

I think you may be running into #8022, which is a known issue. Basically the protos for a bunch of libraries are built, even when the client libraries are not enabled.

I used your command in this Dockerfile to reproduce the problem. Then I squinted at the output of building the Dockerfile.

I see it builds targets like google_cloud_cpp_bigtable_protos, google_cloud_cpp_cloud_bigquery_protos, etc. These are just the protos for those libraries. (The actual client libraries are google_cloud_cpp_bigtable. google_cloud_cpp_bigquery, etc.).

So, obviously it is not great, that you are building extra protos at this time. But at least you know that you are doing the right thing.

Also, thank you for opening up this discussion. It helps us prioritize what issues we should work on next.

[dbolduc]

I hacked together this patch to stop building the extra protos: dbolduc@a0a6b28

I would NOT recommend that you use it in the long term, but if build times are a major problem, it can help.

[aks-zero]

Hi Darren,
Thanks for your checking this and also for your quick reply.

You are right. Most of the libraries out of 74 (except google_cloud_cpp_pubsub, google_cloud_cpp_common and google_cloud_cpp_grpc_utils) are protos libraries.
Will you be able to merge your code, so that we'll pick it next time we upgrade ours?

If you don't mind could you also please tell briefly what's difference between the real library and its protos version e.g. google_cloud_cpp_pubsub vs google_cloud_cpp_pubsub_protos. Does the protos library contain the symbols of source files generated from proto file definitions and the other one contains all client business logic?

Regards,
Ajay

[dbolduc]

Will you be able to merge your code, so that we'll pick it next time we upgrade ours?

I can't merge the code I linked. It is a hack to fix your exact problem (building pubsub only). It is not a general solution to #8022 which would need to work for all permutations of enabled libraries.

I can prioritize the issue though.

Does the protos library contain the symbols of source files generated from proto file definitions and the other one contains all client business logic?

You are exactly correct on the difference between google_cloud_cpp_foo and google_cloud_cpp_foo_protos. I couldn't have said it better myself.

[aks-zero]

Thanks for your reply. That helped.
I have another related question. The reporter of the issue #8022 has mentioned about disabling gRPC during build.

The proto libraries in external/googleapis/ are always compiled (unless gRPC is completely disabled, but that is Okay).

Is there a way to disable gRPC while building google-cloud-cpp?

We are building and installing gRPC and protobuf from source code prior to google-cloud-cpp. We do that because the application also uses gRPC and a custom proto service definition to perform some voice related operations. So are we installing gRPC twice in two separate libraries?

[dbolduc]

the only library that does not depend on gRPC is storage. So "unless gRPC is completely disabled" == "if you are only building storage". gRPC must be present if you are building any other library (like pubsub).

What you are doing sounds right. You are not installing gRPC twice. google-cloud-cpp looks for the installed gRPC, it does not install gRPC itself. If you were to stop building and installing gRPC from source, the google-cloud-cpp install would fail.

[dbolduc]

FWIW, we fixed #8022 in the latest release. You can now build Pub/Sub without having to build the protos from Bigtable, BigQuery, etc.