gax: Excessive byte[] allocations with large read stream requests

[mina-asham]

Originally opened here: googleapis/java-bigtable#2480 then noticed it's not the right repo, so moving here for a fix.

Hi, recently we've been using the BigtableDataClient to send large requests using the readRows streaming method, we observed that with some large requests our CPUs increase to 100%, upon profiling it was mostly the GC running, running an allocation profile showed the following:

Which is coming from: com.google.api.gax.rpc.StateCheckingResponseObserver#onResponse:

public final void onResponse(V response) {
  Preconditions.checkState(!isClosed, getClass() + " received a response after being closed.");
  onResponseImpl(response);
}

The string concatenation is common to be optimized into a StringBuilder by the JVM, so there is a ton of allocation coming from this bit: getClass() + " received a response after being closed.", but the exception is not being thrown here.

Pre-computing the error message every time is an issue, we need to only do that if the exception will be thrown.

Environment details

1. Specify the API at the beginning of the title.
2. OS type and version: Ubuntu 22
3. Java version: OpenJDK17
4. bigtable version(s): 2.60.0

Steps to reproduce

1. Create a BigtableDataClient
2. In a loop send massive requests (1K+ keys)
3. Profile the JVM observe the time and allocations

Code example

BigtableDataSettings settings = BigtableDataSettings.newBuilder()
        .setMetricsProvider(NoopMetricsProvider.INSTANCE)
        .setProjectId("my-project")
        .setInstanceId("my-instance")
        .setAppProfileId("my-profile")
        .build();

BigtableDataClient client = BigtableDataClient.create(settings);
List<String> keys = List.of(); // 1K+ keys that exist
while (!Thread.interrupted()) {
    Query query = Query.create("my-table");
    keys.forEach(query::rowKey);
    client.readRows(query).stream().count(); // Just to consume the stream
}

Stack trace

See the screenshot above

External references such as API reference guides

N/A

Any additional information below

N/A

[mina-asham]

Given how small the change is, I have opened a PR for this: #3622